2023-06-23T01:04:21.283 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-23T01:04:21.284 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/orig.config.yaml 2023-06-23T01:04:21.337 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T01:04:21.432 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945 branch: wip-yuri4-testing-2023-06-21-0846-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} email: yweinste@redhat.com first_in_suite: false job_id: '7311945' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-06-21-0846-quincy ceph: cephfs: ec_profile: - disabled conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 client.0: debug client: 20 debug objectcacher: 20 debug objecter: 20 client.1: debug client: 20 debug objectcacher: 20 debug objecter: 20 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 mon osd backfillfull ratio: 0.6 mon osd full ratio: 0.7 mon osd nearfull ratio: 0.6 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 memstore device bytes: 200000000 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 1.0 osd mon report interval: 5 osd objectstore: memstore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL - PG_RECOVERY_FULL - PG_DEGRADED sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-06-21-0846-quincy sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9037 sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2023-06-21-0846-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c targets: smithi067.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSI9ftZPlRt+IDHUi5hLfCxz7QLJGWQzw8RXkHRFcGsdhLvRaI9dDGsCupQX8AnPEPQNjqvtNkeAzBBPaJrU6mukCrWlcP5De7lGHelE7UA8J1IvH79bsdvgSWpjp3ed5NZVF6A0urzpmxol78o4MUZMibWHiHB+cHw+i18aYXMwKI5H6NkeiqGmD8y0PH5B8rM6vYJg5jVctgUIg/mM+1GV2ZsRKuIQmdiE+6VmHN8aTg4PLF/DWBX2fB2O4j867QZnZUyKaboSKtJgCf8V7FOvYSlewnDvr+yDEwVeXfLxgHR6dLxn+Vl/erM1IIFyqtjJHXASH0J1rOCqENj2g+N5fHaMGGmyTBtjza0h5lVBUSrxkg7BwqzR6xuo/3yoj88cDBrvdcq6f8w6SNcngV35J0dh1x8vERPrjjSn+p10IKVMKkDreofHWSNGsA9CxskVD2dB+40gSsGzQs/ngcwtihMwarIdnHHSAgve3Xt9+vSqrroi44BLzgpooNQss= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBED64u4gjLzpTDT6XA3p9m/odRaKWL8FincvF3djpWK1PElxZKPfNAr+gOw0bHvTOWBCdhDdQlZfrX7eCSWhduQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_full teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-22_15:25:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-23T01:04:21.433 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa; will attempt to use it 2023-06-23T01:04:21.434 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks 2023-06-23T01:04:21.434 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-23T01:04:21.436 INFO:teuthology.task.internal:Checking packages... 2023-06-23T01:04:21.464 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '42ddc46e6e6582d8e29a5a982a81886fe028be2c' 2023-06-23T01:04:21.465 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-23T01:04:21.465 INFO:teuthology.packaging:ref: None 2023-06-23T01:04:21.466 INFO:teuthology.packaging:tag: None 2023-06-23T01:04:21.466 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T01:04:21.466 INFO:teuthology.packaging:sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:04:21.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T01:04:21.605 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-719.g42ddc46e 2023-06-23T01:04:21.606 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-23T01:04:21.612 INFO:teuthology.task.internal:no buildpackages task found 2023-06-23T01:04:21.612 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-23T01:04:21.650 INFO:teuthology.task.internal:Saving configuration 2023-06-23T01:04:21.667 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-23T01:04:21.683 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:04:21.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:59:10.563533', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDYC+U+ZpQ6Z9IM4G7sSn8X7YP/cUEZVDVk1VrC309uHpyXkzRKNihV7z0tasIkOPtR60tX5WJ3vRJabdcrgIKc='} 2023-06-23T01:04:21.737 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:59:10.565178', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBED64u4gjLzpTDT6XA3p9m/odRaKWL8FincvF3djpWK1PElxZKPfNAr+gOw0bHvTOWBCdhDdQlZfrX7eCSWhduQ='} 2023-06-23T01:04:21.737 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-23T01:04:21.745 INFO:teuthology.task.internal:roles: ubuntu@smithi067.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-06-23T01:04:21.745 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-06-23T01:04:21.746 INFO:teuthology.run_tasks:Running task console_log... 2023-06-23T01:04:21.851 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f615438e040>, signals=[15]) 2023-06-23T01:04:21.852 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-23T01:04:21.859 INFO:teuthology.task.internal:Opening connections... 2023-06-23T01:04:21.859 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:04:21.865 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:04:21.936 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:04:21.938 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:04:22.006 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-23T01:04:22.013 DEBUG:teuthology.orchestra.run.smithi067:> uname -m 2023-06-23T01:04:22.031 INFO:teuthology.orchestra.run.smithi067.stdout:x86_64 2023-06-23T01:04:22.031 DEBUG:teuthology.orchestra.run.smithi067:> cat /etc/os-release 2023-06-23T01:04:22.087 INFO:teuthology.orchestra.run.smithi067.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T01:04:22.088 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T01:04:22.088 INFO:teuthology.orchestra.run.smithi067.stdout:ID="rhel" 2023-06-23T01:04:22.088 INFO:teuthology.orchestra.run.smithi067.stdout:ID_LIKE="fedora" 2023-06-23T01:04:22.088 INFO:teuthology.orchestra.run.smithi067.stdout:VERSION_ID="8.4" 2023-06-23T01:04:22.088 INFO:teuthology.orchestra.run.smithi067.stdout:PLATFORM_ID="platform:el8" 2023-06-23T01:04:22.088 INFO:teuthology.orchestra.run.smithi067.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T01:04:22.089 INFO:teuthology.orchestra.run.smithi067.stdout:ANSI_COLOR="0;31" 2023-06-23T01:04:22.089 INFO:teuthology.orchestra.run.smithi067.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T01:04:22.089 INFO:teuthology.orchestra.run.smithi067.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T01:04:22.089 INFO:teuthology.orchestra.run.smithi067.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T01:04:22.089 INFO:teuthology.orchestra.run.smithi067.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T01:04:22.089 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:04:22.089 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T01:04:22.090 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T01:04:22.090 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T01:04:22.090 INFO:teuthology.orchestra.run.smithi067.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T01:04:22.091 INFO:teuthology.lock.ops:Updating smithi067.front.sepia.ceph.com on lock server 2023-06-23T01:04:22.113 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2023-06-23T01:04:22.131 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2023-06-23T01:04:22.132 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2023-06-23T01:04:22.187 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T01:04:22.187 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T01:04:22.187 INFO:teuthology.orchestra.run.smithi193.stdout:ID="rhel" 2023-06-23T01:04:22.187 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="fedora" 2023-06-23T01:04:22.188 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="8.4" 2023-06-23T01:04:22.188 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el8" 2023-06-23T01:04:22.188 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T01:04:22.188 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2023-06-23T01:04:22.188 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T01:04:22.188 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T01:04:22.188 INFO:teuthology.orchestra.run.smithi193.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T01:04:22.189 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T01:04:22.189 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:04:22.189 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T01:04:22.189 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T01:04:22.189 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T01:04:22.189 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T01:04:22.190 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2023-06-23T01:04:22.213 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-23T01:04:22.222 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-23T01:04:22.249 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-23T01:04:22.250 DEBUG:teuthology.orchestra.run.smithi067:> test '!' -e /home/ubuntu/cephtest 2023-06-23T01:04:22.252 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2023-06-23T01:04:22.267 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-23T01:04:22.273 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-23T01:04:22.273 DEBUG:teuthology.orchestra.run.smithi067:> test -z $(ls -A /var/lib/ceph) 2023-06-23T01:04:22.309 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2023-06-23T01:04:22.323 INFO:teuthology.orchestra.run.smithi067.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T01:04:22.325 INFO:teuthology.orchestra.run.smithi193.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T01:04:22.326 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-23T01:04:22.382 INFO:teuthology.run_tasks:Running task kernel... 2023-06-23T01:04:22.397 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-06-23T01:04:22.398 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-06-23T01:04:22.398 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-06-23T01:04:22.398 DEBUG:teuthology.orchestra.run.smithi067:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T01:04:22.399 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T01:04:22.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:04:22.415 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2023-06-23T01:04:22.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:04:22.416 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2023-06-23T01:04:22.471 INFO:teuthology.orchestra.run.smithi193.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T01:04:22.471 INFO:teuthology.task.kernel:Running kernel on smithi193: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T01:04:22.471 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum install -y kernel 2023-06-23T01:04:22.473 INFO:teuthology.orchestra.run.smithi067.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T01:04:22.473 INFO:teuthology.task.kernel:Running kernel on smithi067: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T01:04:22.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum install -y kernel 2023-06-23T01:04:23.917 INFO:teuthology.orchestra.run.smithi067.stdout:Updating Subscription Management repositories. 2023-06-23T01:04:23.918 INFO:teuthology.orchestra.run.smithi067.stdout:Unable to read consumer identity 2023-06-23T01:04:23.958 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2023-06-23T01:04:23.959 INFO:teuthology.orchestra.run.smithi193.stdout:Unable to read consumer identity 2023-06-23T01:04:26.993 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2023-06-23T01:04:27.680 INFO:teuthology.orchestra.run.smithi193.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T01:04:29.683 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 5.9 MB/s | 16 MB 00:02 2023-06-23T01:04:30.942 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 4.8 MB/s | 16 MB 00:03 2023-06-23T01:04:33.108 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-06-23T01:04:33.807 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T01:04:33.808 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T01:04:33.808 INFO:teuthology.orchestra.run.smithi067.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T01:04:33.966 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-06-23T01:04:33.967 INFO:teuthology.orchestra.run.smithi067.stdout:Nothing to do. 2023-06-23T01:04:33.967 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-06-23T01:04:34.101 DEBUG:teuthology.orchestra.run.smithi067:> echo no | sudo yum reinstall kernel || true 2023-06-23T01:04:34.280 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-06-23T01:04:34.467 INFO:teuthology.orchestra.run.smithi067.stdout:Updating Subscription Management repositories. 2023-06-23T01:04:34.467 INFO:teuthology.orchestra.run.smithi067.stdout:Unable to read consumer identity 2023-06-23T01:04:34.780 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:01 ago on Fri 23 Jun 2023 01:04:33 AM UTC. 2023-06-23T01:04:34.911 INFO:teuthology.orchestra.run.smithi067.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T01:04:34.911 INFO:teuthology.orchestra.run.smithi067.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:04:34.911 INFO:teuthology.orchestra.run.smithi067.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:04:34.936 INFO:teuthology.orchestra.run.smithi067.stderr:Error: No packages marked for reinstall. 2023-06-23T01:04:34.948 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T01:04:34.948 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T01:04:34.949 INFO:teuthology.orchestra.run.smithi193.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T01:04:34.993 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum reinstall -y kernel || true 2023-06-23T01:04:35.097 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2023-06-23T01:04:35.098 INFO:teuthology.orchestra.run.smithi193.stdout:Nothing to do. 2023-06-23T01:04:35.099 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2023-06-23T01:04:35.202 DEBUG:teuthology.orchestra.run.smithi193:> echo no | sudo yum reinstall kernel || true 2023-06-23T01:04:35.359 INFO:teuthology.orchestra.run.smithi067.stdout:Updating Subscription Management repositories. 2023-06-23T01:04:35.359 INFO:teuthology.orchestra.run.smithi067.stdout:Unable to read consumer identity 2023-06-23T01:04:35.546 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2023-06-23T01:04:35.546 INFO:teuthology.orchestra.run.smithi193.stdout:Unable to read consumer identity 2023-06-23T01:04:35.660 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 01:04:33 AM UTC. 2023-06-23T01:04:35.802 INFO:teuthology.orchestra.run.smithi067.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T01:04:35.802 INFO:teuthology.orchestra.run.smithi067.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:04:35.802 INFO:teuthology.orchestra.run.smithi067.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:04:35.811 INFO:teuthology.orchestra.run.smithi067.stderr:Error: No packages marked for reinstall. 2023-06-23T01:04:35.837 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:01 ago on Fri 23 Jun 2023 01:04:34 AM UTC. 2023-06-23T01:04:35.864 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:04:35.961 INFO:teuthology.orchestra.run.smithi193.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T01:04:35.961 INFO:teuthology.orchestra.run.smithi193.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:04:35.961 INFO:teuthology.orchestra.run.smithi193.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:04:35.993 INFO:teuthology.orchestra.run.smithi193.stderr:Error: No packages marked for reinstall. 2023-06-23T01:04:35.994 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:35.995 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:35.995 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:35.996 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T01:04:35.996 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T01:04:35.996 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-06-23T01:04:35.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi067.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T01:04:35.997 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:04:36.042 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum reinstall -y kernel || true 2023-06-23T01:04:36.127 INFO:teuthology.orchestra.run.smithi067.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:36.128 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:36.128 DEBUG:teuthology.orchestra.run.smithi067:> sudo rpm -qi grub2-tools 2023-06-23T01:04:36.156 INFO:teuthology.orchestra.run.smithi067.stdout:Name : grub2-tools 2023-06-23T01:04:36.157 INFO:teuthology.orchestra.run.smithi067.stdout:Epoch : 1 2023-06-23T01:04:36.157 INFO:teuthology.orchestra.run.smithi067.stdout:Version : 2.02 2023-06-23T01:04:36.157 INFO:teuthology.orchestra.run.smithi067.stdout:Release : 99.el8_4.1 2023-06-23T01:04:36.157 INFO:teuthology.orchestra.run.smithi067.stdout:Architecture: x86_64 2023-06-23T01:04:36.157 INFO:teuthology.orchestra.run.smithi067.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T01:04:36.157 INFO:teuthology.orchestra.run.smithi067.stdout:Group : System Environment/Base 2023-06-23T01:04:36.157 INFO:teuthology.orchestra.run.smithi067.stdout:Size : 9264631 2023-06-23T01:04:36.158 INFO:teuthology.orchestra.run.smithi067.stdout:License : GPLv3+ 2023-06-23T01:04:36.158 INFO:teuthology.orchestra.run.smithi067.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T01:04:36.158 INFO:teuthology.orchestra.run.smithi067.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T01:04:36.158 INFO:teuthology.orchestra.run.smithi067.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T01:04:36.158 INFO:teuthology.orchestra.run.smithi067.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T01:04:36.158 INFO:teuthology.orchestra.run.smithi067.stdout:Relocations : (not relocatable) 2023-06-23T01:04:36.158 INFO:teuthology.orchestra.run.smithi067.stdout:Packager : Red Hat, Inc. 2023-06-23T01:04:36.158 INFO:teuthology.orchestra.run.smithi067.stdout:Vendor : Red Hat, Inc. 2023-06-23T01:04:36.159 INFO:teuthology.orchestra.run.smithi067.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T01:04:36.159 INFO:teuthology.orchestra.run.smithi067.stdout:Summary : Support tools for GRUB. 2023-06-23T01:04:36.159 INFO:teuthology.orchestra.run.smithi067.stdout:Description : 2023-06-23T01:04:36.159 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:04:36.159 INFO:teuthology.orchestra.run.smithi067.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T01:04:36.159 INFO:teuthology.orchestra.run.smithi067.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T01:04:36.159 INFO:teuthology.orchestra.run.smithi067.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T01:04:36.160 INFO:teuthology.orchestra.run.smithi067.stdout:hardware devices. 2023-06-23T01:04:36.160 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:04:36.160 INFO:teuthology.orchestra.run.smithi067.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T01:04:36.161 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T01:04:36.161 INFO:teuthology.task.kernel:Updating grub on smithi067 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:36.161 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T01:04:36.385 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2023-06-23T01:04:36.386 INFO:teuthology.orchestra.run.smithi193.stdout:Unable to read consumer identity 2023-06-23T01:04:36.672 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 01:04:34 AM UTC. 2023-06-23T01:04:36.802 INFO:teuthology.orchestra.run.smithi193.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T01:04:36.802 INFO:teuthology.orchestra.run.smithi193.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:04:36.802 INFO:teuthology.orchestra.run.smithi193.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:04:36.823 INFO:teuthology.orchestra.run.smithi193.stderr:Error: No packages marked for reinstall. 2023-06-23T01:04:36.872 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:04:36.997 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:36.998 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:36.998 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:36.998 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T01:04:36.998 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T01:04:36.998 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-06-23T01:04:36.998 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi193.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T01:04:36.999 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:04:37.123 INFO:teuthology.orchestra.run.smithi193.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:37.123 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:37.123 DEBUG:teuthology.orchestra.run.smithi193:> sudo rpm -qi grub2-tools 2023-06-23T01:04:37.151 INFO:teuthology.orchestra.run.smithi193.stdout:Name : grub2-tools 2023-06-23T01:04:37.151 INFO:teuthology.orchestra.run.smithi193.stdout:Epoch : 1 2023-06-23T01:04:37.151 INFO:teuthology.orchestra.run.smithi193.stdout:Version : 2.02 2023-06-23T01:04:37.151 INFO:teuthology.orchestra.run.smithi193.stdout:Release : 99.el8_4.1 2023-06-23T01:04:37.151 INFO:teuthology.orchestra.run.smithi193.stdout:Architecture: x86_64 2023-06-23T01:04:37.152 INFO:teuthology.orchestra.run.smithi193.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T01:04:37.152 INFO:teuthology.orchestra.run.smithi193.stdout:Group : System Environment/Base 2023-06-23T01:04:37.152 INFO:teuthology.orchestra.run.smithi193.stdout:Size : 9264631 2023-06-23T01:04:37.152 INFO:teuthology.orchestra.run.smithi193.stdout:License : GPLv3+ 2023-06-23T01:04:37.152 INFO:teuthology.orchestra.run.smithi193.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T01:04:37.152 INFO:teuthology.orchestra.run.smithi193.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T01:04:37.152 INFO:teuthology.orchestra.run.smithi193.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T01:04:37.153 INFO:teuthology.orchestra.run.smithi193.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T01:04:37.153 INFO:teuthology.orchestra.run.smithi193.stdout:Relocations : (not relocatable) 2023-06-23T01:04:37.153 INFO:teuthology.orchestra.run.smithi193.stdout:Packager : Red Hat, Inc. 2023-06-23T01:04:37.153 INFO:teuthology.orchestra.run.smithi193.stdout:Vendor : Red Hat, Inc. 2023-06-23T01:04:37.153 INFO:teuthology.orchestra.run.smithi193.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T01:04:37.153 INFO:teuthology.orchestra.run.smithi193.stdout:Summary : Support tools for GRUB. 2023-06-23T01:04:37.153 INFO:teuthology.orchestra.run.smithi193.stdout:Description : 2023-06-23T01:04:37.153 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:04:37.154 INFO:teuthology.orchestra.run.smithi193.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T01:04:37.154 INFO:teuthology.orchestra.run.smithi193.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T01:04:37.154 INFO:teuthology.orchestra.run.smithi193.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T01:04:37.154 INFO:teuthology.orchestra.run.smithi193.stdout:hardware devices. 2023-06-23T01:04:37.154 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:04:37.154 INFO:teuthology.orchestra.run.smithi193.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T01:04:37.158 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T01:04:37.158 INFO:teuthology.task.kernel:Updating grub on smithi193 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:37.158 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T01:04:37.305 INFO:teuthology.orchestra.run.smithi067.stderr:Generating grub configuration file ... 2023-06-23T01:04:38.312 INFO:teuthology.orchestra.run.smithi193.stderr:Generating grub configuration file ... 2023-06-23T01:04:39.649 INFO:teuthology.orchestra.run.smithi067.stderr:done 2023-06-23T01:04:39.651 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2023-06-23T01:04:39.666 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.6R8wggJ1s0 2023-06-23T01:04:39.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6R8wggJ1s0 2023-06-23T01:04:39.736 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.6R8wggJ1s0 2023-06-23T01:04:39.896 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.6R8wggJ1s0 is 6KB 2023-06-23T01:04:39.944 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.6R8wggJ1s0 2023-06-23T01:04:39.958 DEBUG:teuthology.orchestra.run.smithi067:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T01:04:40.033 INFO:teuthology.orchestra.run.smithi067.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T01:04:40.034 INFO:teuthology.orchestra.run.smithi067.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T01:04:40.034 INFO:teuthology.orchestra.run.smithi067.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T01:04:40.034 INFO:teuthology.orchestra.run.smithi067.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T01:04:40.035 DEBUG:teuthology.orchestra.run.smithi067:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:40.178 DEBUG:teuthology.orchestra.run.smithi067:> sudo shutdown -r now 2023-06-23T01:04:40.512 INFO:teuthology.orchestra.run.smithi193.stderr:done 2023-06-23T01:04:40.514 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2023-06-23T01:04:40.527 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.raf8VSnPuB 2023-06-23T01:04:40.528 DEBUG:teuthology.orchestra.run.smithi193:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.raf8VSnPuB 2023-06-23T01:04:40.597 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.raf8VSnPuB 2023-06-23T01:04:40.754 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.raf8VSnPuB is 6KB 2023-06-23T01:04:40.803 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.raf8VSnPuB 2023-06-23T01:04:40.816 DEBUG:teuthology.orchestra.run.smithi193:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T01:04:40.898 INFO:teuthology.orchestra.run.smithi193.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T01:04:40.898 INFO:teuthology.orchestra.run.smithi193.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T01:04:40.899 INFO:teuthology.orchestra.run.smithi193.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T01:04:40.899 INFO:teuthology.orchestra.run.smithi193.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T01:04:40.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:04:41.077 DEBUG:teuthology.orchestra.run.smithi193:> sudo shutdown -r now 2023-06-23T01:05:10.208 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-06-23T01:05:10.208 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-06-23T01:05:10.209 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:05:11.081 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-06-23T01:05:11.081 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-06-23T01:05:11.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:05:28.692 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2023-06-23T01:05:29.460 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.193 2023-06-23T01:05:58.722 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-06-23T01:05:58.723 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:05:59.462 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-06-23T01:05:59.463 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:06:17.059 DEBUG:teuthology.orchestra.run.smithi193:> true 2023-06-23T01:06:17.072 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.67 2023-06-23T01:06:17.287 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-06-23T01:06:17.288 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T01:06:17.288 DEBUG:teuthology.orchestra.run.smithi193:> uname -r 2023-06-23T01:06:17.343 INFO:teuthology.orchestra.run.smithi193.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T01:06:17.343 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:06:17.344 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T01:06:17.344 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T01:06:18.345 DEBUG:teuthology.parallel:result is None 2023-06-23T01:06:47.073 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-06-23T01:06:47.074 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:06:47.403 DEBUG:teuthology.orchestra.run.smithi067:> true 2023-06-23T01:06:47.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-06-23T01:06:47.710 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T01:06:47.710 DEBUG:teuthology.orchestra.run.smithi067:> uname -r 2023-06-23T01:06:47.726 INFO:teuthology.orchestra.run.smithi067.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T01:06:47.726 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:06:47.727 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T01:06:47.727 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T01:06:48.728 DEBUG:teuthology.parallel:result is None 2023-06-23T01:06:48.729 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-23T01:06:48.736 INFO:teuthology.task.internal:Creating test directory... 2023-06-23T01:06:48.737 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T01:06:48.740 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T01:06:48.758 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-23T01:06:48.799 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-23T01:06:48.843 INFO:teuthology.task.internal:Creating archive directory... 2023-06-23T01:06:48.844 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T01:06:48.847 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T01:06:48.949 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-23T01:06:48.956 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-23T01:06:48.957 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T01:06:48.959 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T01:06:49.053 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:06:49.057 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:06:49.074 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:06:49.075 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:06:49.077 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-23T01:06:49.085 INFO:teuthology.task.internal:Configuring sudo... 2023-06-23T01:06:49.085 DEBUG:teuthology.orchestra.run.smithi067:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T01:06:49.088 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T01:06:49.114 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-23T01:06:49.123 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-23T01:06:49.124 DEBUG:teuthology.orchestra.run.smithi067:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T01:06:49.156 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T01:06:49.171 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:06:49.258 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:06:49.341 DEBUG:teuthology.orchestra.run.smithi067:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:06:49.388 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:06:49.424 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:06:49.425 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T01:06:49.492 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:06:49.536 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:06:49.576 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:06:49.657 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:06:49.696 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:06:49.696 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T01:06:49.760 DEBUG:teuthology.orchestra.run.smithi067:> sudo service rsyslog restart 2023-06-23T01:06:49.763 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2023-06-23T01:06:49.810 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:06:49.855 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:06:50.297 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-23T01:06:50.305 INFO:teuthology.task.internal:Starting timer... 2023-06-23T01:06:50.305 INFO:teuthology.run_tasks:Running task pcp... 2023-06-23T01:06:50.352 INFO:teuthology.run_tasks:Running task selinux... 2023-06-23T01:06:50.391 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-06-23T01:06:50.392 DEBUG:teuthology.orchestra.run.smithi067:> sudo service auditd rotate 2023-06-23T01:06:50.450 INFO:teuthology.orchestra.run.smithi067.stdout:Rotating logs: [ OK ] 2023-06-23T01:06:50.451 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2023-06-23T01:06:50.512 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: [ OK ] 2023-06-23T01:06:50.514 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-23T01:06:50.514 DEBUG:teuthology.orchestra.run.smithi067:> /usr/sbin/getenforce 2023-06-23T01:06:50.536 INFO:teuthology.orchestra.run.smithi067.stdout:Permissive 2023-06-23T01:06:50.536 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2023-06-23T01:06:50.555 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2023-06-23T01:06:50.556 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi067.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2023-06-23T01:06:50.556 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:06:50.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:06:50.603 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:06:50.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:06:50.630 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-23T01:06:50.630 DEBUG:teuthology.orchestra.run.smithi067:> sudo /usr/sbin/setenforce permissive 2023-06-23T01:06:50.670 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2023-06-23T01:06:50.696 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-23T01:06:50.706 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-06-23T01:06:50.713 INFO:teuthology.repo_utils:Fetching main from origin 2023-06-23T01:06:50.859 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-23T01:06:50.887 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-23T01:06:50.889 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi067.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-23T01:17:09.361 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi067.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2023-06-23T01:17:09.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi067.front.sepia.ceph.com' 2023-06-23T01:17:09.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi067.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:17:09.440 DEBUG:teuthology.orchestra.run.smithi067:> true 2023-06-23T01:17:09.508 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2023-06-23T01:17:09.509 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:17:09.578 DEBUG:teuthology.orchestra.run.smithi193:> true 2023-06-23T01:17:09.648 INFO:teuthology.run_tasks:Running task clock... 2023-06-23T01:17:09.661 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-23T01:17:09.662 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T01:17:09.663 DEBUG:teuthology.orchestra.run.smithi067:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:17:09.665 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T01:17:09.666 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:17:09.695 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T01:17:09.711 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T01:17:09.717 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T01:17:09.731 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T01:17:09.748 INFO:teuthology.orchestra.run.smithi067.stderr:sudo: ntpd: command not found 2023-06-23T01:17:09.759 INFO:teuthology.orchestra.run.smithi067.stdout:506 Cannot talk to daemon 2023-06-23T01:17:09.767 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2023-06-23T01:17:09.773 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T01:17:09.778 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2023-06-23T01:17:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T01:17:09.792 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T01:17:09.806 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T01:17:09.855 INFO:teuthology.orchestra.run.smithi067.stderr:bash: ntpq: command not found 2023-06-23T01:17:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:210 Number of sources = 4 2023-06-23T01:17:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:17:09.858 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2023-06-23T01:17:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:17:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:17:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:17:09.859 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:17:09.881 INFO:teuthology.orchestra.run.smithi193.stderr:bash: ntpq: command not found 2023-06-23T01:17:09.884 INFO:teuthology.orchestra.run.smithi193.stdout:210 Number of sources = 4 2023-06-23T01:17:09.884 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:17:09.884 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2023-06-23T01:17:09.884 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:17:09.884 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:17:09.885 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:17:09.885 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:17:09.885 INFO:teuthology.run_tasks:Running task install... 2023-06-23T01:17:09.893 DEBUG:teuthology.task.install:project ceph 2023-06-23T01:17:09.894 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}} 2023-06-23T01:17:09.894 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-23T01:17:09.894 INFO:teuthology.task.install:Using flavor: default 2023-06-23T01:17:09.906 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-23T01:17:09.906 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-23T01:17:09.907 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T01:17:09.908 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:17:09.910 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T01:17:09.911 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:17:10.128 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T01:17:10.128 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T01:17:10.134 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T01:17:10.134 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T01:17:10.371 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T01:17:10.372 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:17:10.372 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T01:17:10.373 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T01:17:10.373 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:17:10.373 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T01:17:10.399 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T01:17:10.400 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T01:17:10.401 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T01:17:10.402 DEBUG:teuthology.orchestra.run.smithi067:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T01:17:10.465 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T01:17:10.468 DEBUG:teuthology.orchestra.run.smithi067:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T01:17:10.539 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T01:17:10.544 DEBUG:teuthology.orchestra.run.smithi067:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T01:17:10.603 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2023-06-23T01:17:10.604 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2023-06-23T01:17:10.609 INFO:teuthology.orchestra.run.smithi067.stdout:check_obsoletes = 1 2023-06-23T01:17:10.610 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum clean all 2023-06-23T01:17:10.998 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2023-06-23T01:17:11.021 INFO:teuthology.orchestra.run.smithi067.stdout:Updating Subscription Management repositories. 2023-06-23T01:17:11.389 INFO:teuthology.orchestra.run.smithi193.stdout:44 files removed 2023-06-23T01:17:11.422 INFO:teuthology.orchestra.run.smithi067.stdout:44 files removed 2023-06-23T01:17:11.439 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T01:17:11.473 DEBUG:teuthology.orchestra.run.smithi067:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T01:17:11.792 INFO:teuthology.orchestra.run.smithi193.stdout:Updating Subscription Management repositories. 2023-06-23T01:17:11.845 INFO:teuthology.orchestra.run.smithi067.stdout:Updating Subscription Management repositories. 2023-06-23T01:17:12.872 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 143 kB/s | 78 kB 00:00 2023-06-23T01:17:12.968 INFO:teuthology.orchestra.run.smithi067.stdout:ceph packages for x86_64 135 kB/s | 78 kB 00:00 2023-06-23T01:17:13.579 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 14 kB/s | 9.2 kB 00:00 2023-06-23T01:17:13.591 INFO:teuthology.orchestra.run.smithi067.stdout:ceph noarch packages 16 kB/s | 9.2 kB 00:00 2023-06-23T01:17:14.305 INFO:teuthology.orchestra.run.smithi067.stdout:ceph source packages 2.1 kB/s | 1.4 kB 00:00 2023-06-23T01:17:14.345 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 2.0 kB/s | 1.4 kB 00:00 2023-06-23T01:17:16.122 INFO:teuthology.orchestra.run.smithi067.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2023-06-23T01:17:16.213 INFO:teuthology.orchestra.run.smithi193.stdout:Copr repo for python3-asyncssh owned by ceph 1.9 kB/s | 3.5 kB 00:01 2023-06-23T01:17:21.408 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 3.0 MB/s | 16 MB 00:05 2023-06-23T01:17:24.637 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 4.2 MB/s | 24 kB 00:00 2023-06-23T01:17:24.996 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 118 MB/s | 33 MB 00:00 2023-06-23T01:17:30.149 INFO:teuthology.orchestra.run.smithi193.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 123 MB/s | 38 MB 00:00 2023-06-23T01:17:38.365 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:17:38.366 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:17:38.376 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T01:17:38.377 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T01:17:38.378 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T01:17:38.379 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T01:17:38.381 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T01:17:38.383 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T01:17:38.384 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T01:17:38.884 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2023-06-23T01:17:38.895 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================================================== 2023-06-23T01:17:38.895 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2023-06-23T01:17:38.896 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================================================== 2023-06-23T01:17:38.896 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2023-06-23T01:17:38.896 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T01:17:38.896 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T01:17:38.896 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T01:17:38.896 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T01:17:38.896 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T01:17:38.897 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T01:17:38.897 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T01:17:38.897 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T01:17:38.897 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T01:17:38.897 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T01:17:38.897 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:17:38.897 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T01:17:38.897 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T01:17:38.898 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T01:17:38.898 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T01:17:38.898 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T01:17:38.898 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T01:17:38.899 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T01:17:38.899 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T01:17:38.899 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T01:17:38.899 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T01:17:38.899 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T01:17:38.899 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T01:17:38.899 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T01:17:38.899 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T01:17:38.900 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T01:17:38.900 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T01:17:38.901 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T01:17:38.901 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T01:17:38.901 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T01:17:38.901 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:17:38.901 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T01:17:38.901 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T01:17:38.901 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2023-06-23T01:17:38.901 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:17:38.902 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T01:17:38.902 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2023-06-23T01:17:38.902 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T01:17:38.902 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T01:17:38.902 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T01:17:38.902 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T01:17:38.902 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T01:17:38.903 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T01:17:38.903 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:17:38.903 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T01:17:38.903 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T01:17:38.903 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T01:17:38.903 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T01:17:38.903 INFO:teuthology.orchestra.run.smithi193.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T01:17:38.903 INFO:teuthology.orchestra.run.smithi193.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T01:17:38.904 INFO:teuthology.orchestra.run.smithi193.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T01:17:38.904 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T01:17:38.905 INFO:teuthology.orchestra.run.smithi193.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T01:17:38.905 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T01:17:38.906 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T01:17:38.906 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T01:17:38.906 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T01:17:38.907 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T01:17:38.907 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T01:17:38.907 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T01:17:38.907 INFO:teuthology.orchestra.run.smithi193.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T01:17:38.908 INFO:teuthology.orchestra.run.smithi193.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T01:17:38.908 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T01:17:38.908 INFO:teuthology.orchestra.run.smithi193.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T01:17:38.909 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T01:17:38.909 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T01:17:38.909 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T01:17:38.910 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T01:17:38.910 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T01:17:38.910 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T01:17:38.910 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T01:17:38.911 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T01:17:38.911 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T01:17:38.911 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T01:17:38.911 INFO:teuthology.orchestra.run.smithi193.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T01:17:38.912 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T01:17:38.912 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T01:17:38.912 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T01:17:38.913 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T01:17:38.913 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T01:17:38.913 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T01:17:38.914 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T01:17:38.914 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T01:17:38.914 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T01:17:38.915 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T01:17:38.915 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T01:17:38.915 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T01:17:38.916 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:17:38.916 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T01:17:38.916 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T01:17:38.916 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T01:17:38.917 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T01:17:38.917 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T01:17:38.917 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T01:17:38.918 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T01:17:38.918 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T01:17:38.918 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T01:17:38.918 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T01:17:38.919 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T01:17:38.919 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T01:17:38.919 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T01:17:38.919 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T01:17:38.920 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T01:17:38.920 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T01:17:38.921 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:17:38.921 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T01:17:38.921 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T01:17:38.922 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T01:17:38.922 INFO:teuthology.orchestra.run.smithi193.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T01:17:38.922 INFO:teuthology.orchestra.run.smithi193.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T01:17:38.923 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T01:17:38.923 INFO:teuthology.orchestra.run.smithi193.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T01:17:38.923 INFO:teuthology.orchestra.run.smithi193.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T01:17:38.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T01:17:38.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T01:17:38.924 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T01:17:38.925 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T01:17:38.925 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T01:17:38.925 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T01:17:38.925 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T01:17:38.926 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T01:17:38.926 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T01:17:38.926 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:17:38.926 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2023-06-23T01:17:38.927 INFO:teuthology.orchestra.run.smithi193.stdout:====================================================================================================================== 2023-06-23T01:17:38.927 INFO:teuthology.orchestra.run.smithi193.stdout:Install 118 Packages 2023-06-23T01:17:38.928 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 2 Packages 2023-06-23T01:17:38.928 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:17:38.929 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 201 M 2023-06-23T01:17:38.929 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2023-06-23T01:17:39.068 INFO:teuthology.orchestra.run.smithi193.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 68 kB/s | 6.4 kB 00:00 2023-06-23T01:17:39.216 INFO:teuthology.orchestra.run.smithi193.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 5.8 MB/s | 873 kB 00:00 2023-06-23T01:17:39.238 INFO:teuthology.orchestra.run.smithi193.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 7.2 MB/s | 162 kB 00:00 2023-06-23T01:17:39.473 INFO:teuthology.orchestra.run.smithi193.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 9.8 MB/s | 2.3 MB 00:00 2023-06-23T01:17:39.510 INFO:teuthology.orchestra.run.smithi193.stdout:(5/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 9.6 MB/s | 5.2 MB 00:00 2023-06-23T01:17:39.624 INFO:teuthology.orchestra.run.smithi193.stdout:(6/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 10 MB/s | 1.6 MB 00:00 2023-06-23T01:17:39.849 INFO:teuthology.orchestra.run.smithi193.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 12 MB/s | 3.9 MB 00:00 2023-06-23T01:17:41.604 INFO:teuthology.orchestra.run.smithi193.stdout:(8/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8. 9.5 MB/s | 17 MB 00:01 2023-06-23T01:17:41.724 INFO:teuthology.orchestra.run.smithi193.stdout:(9/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86_ 7.9 MB/s | 17 MB 00:02 2023-06-23T01:17:41.725 INFO:teuthology.orchestra.run.smithi193.stdout:(10/120): ceph-selinux-17.2.6-719.g42ddc46e.el8 201 kB/s | 24 kB 00:00 2023-06-23T01:17:41.766 INFO:teuthology.orchestra.run.smithi193.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 6.2 MB/s | 255 kB 00:00 2023-06-23T01:17:41.800 INFO:teuthology.orchestra.run.smithi193.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 731 kB/s | 25 kB 00:00 2023-06-23T01:17:41.897 INFO:teuthology.orchestra.run.smithi193.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 8.0 MB/s | 794 kB 00:00 2023-06-23T01:17:41.943 INFO:teuthology.orchestra.run.smithi193.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 3.8 MB/s | 174 kB 00:00 2023-06-23T01:17:41.987 INFO:teuthology.orchestra.run.smithi193.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 2.9 MB/s | 130 kB 00:00 2023-06-23T01:17:42.042 INFO:teuthology.orchestra.run.smithi193.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 9.0 MB/s | 506 kB 00:00 2023-06-23T01:17:42.571 INFO:teuthology.orchestra.run.smithi193.stdout:(17/120): ceph-common-17.2.6-719.g42ddc46e.el8. 6.5 MB/s | 23 MB 00:03 2023-06-23T01:17:42.595 INFO:teuthology.orchestra.run.smithi193.stdout:(18/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 10 MB/s | 5.5 MB 00:00 2023-06-23T01:17:42.622 INFO:teuthology.orchestra.run.smithi193.stdout:(19/120): python3-ceph-argparse-17.2.6-719.g42d 870 kB/s | 45 kB 00:00 2023-06-23T01:17:42.625 INFO:teuthology.orchestra.run.smithi193.stdout:(20/120): python3-ceph-common-17.2.6-719.g42ddc 3.8 MB/s | 115 kB 00:00 2023-06-23T01:17:42.669 INFO:teuthology.orchestra.run.smithi193.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 8.3 MB/s | 371 kB 00:00 2023-06-23T01:17:42.670 INFO:teuthology.orchestra.run.smithi193.stdout:(22/120): python3-cephfs-17.2.6-719.g42ddc46e.e 3.9 MB/s | 190 kB 00:00 2023-06-23T01:17:42.706 INFO:teuthology.orchestra.run.smithi193.stdout:(23/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 9.5 MB/s | 357 kB 00:00 2023-06-23T01:17:42.725 INFO:teuthology.orchestra.run.smithi193.stdout:(24/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.9 MB/s | 107 kB 00:00 2023-06-23T01:17:42.737 INFO:teuthology.orchestra.run.smithi193.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 2.9 MB/s | 91 kB 00:00 2023-06-23T01:17:42.787 INFO:teuthology.orchestra.run.smithi193.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 3.4 MB/s | 173 kB 00:00 2023-06-23T01:17:42.833 INFO:teuthology.orchestra.run.smithi193.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 513 kB/s | 23 kB 00:00 2023-06-23T01:17:42.870 INFO:teuthology.orchestra.run.smithi193.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 3.2 MB/s | 121 kB 00:00 2023-06-23T01:17:43.051 INFO:teuthology.orchestra.run.smithi193.stdout:(29/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 9.4 MB/s | 1.7 MB 00:00 2023-06-23T01:17:43.160 INFO:teuthology.orchestra.run.smithi193.stdout:(30/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 8.0 MB/s | 3.5 MB 00:00 2023-06-23T01:17:43.239 INFO:teuthology.orchestra.run.smithi193.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 3.0 MB/s | 238 kB 00:00 2023-06-23T01:17:43.310 INFO:teuthology.orchestra.run.smithi193.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 667 kB/s | 47 kB 00:00 2023-06-23T01:17:43.380 INFO:teuthology.orchestra.run.smithi193.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 207 kB/s | 14 kB 00:00 2023-06-23T01:17:43.459 INFO:teuthology.orchestra.run.smithi193.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 3.3 MB/s | 264 kB 00:00 2023-06-23T01:17:43.532 INFO:teuthology.orchestra.run.smithi193.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.1 MB/s | 80 kB 00:00 2023-06-23T01:17:43.593 INFO:teuthology.orchestra.run.smithi193.stdout:(36/120): ceph-mgr-diskprediction-local-17.2.6- 14 MB/s | 7.4 MB 00:00 2023-06-23T01:17:43.600 INFO:teuthology.orchestra.run.smithi193.stdout:(37/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 262 kB/s | 18 kB 00:00 2023-06-23T01:17:43.609 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:43.633 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:43.634 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:43.634 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:43.705 INFO:teuthology.orchestra.run.smithi193.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 565 kB/s | 59 kB 00:00 2023-06-23T01:17:43.705 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:43.706 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:43.740 INFO:teuthology.orchestra.run.smithi193.stdout:(39/120): gperftools-libs-2.7-9.el8.x86_64.rpm 2.0 MB/s | 306 kB 00:00 2023-06-23T01:17:43.741 INFO:teuthology.orchestra.run.smithi193.stdout:(40/120): libunwind-1.3.1-3.el8.x86_64.rpm 2.0 MB/s | 75 kB 00:00 2023-06-23T01:17:43.742 INFO:teuthology.orchestra.run.smithi193.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T01:17:43.759 INFO:teuthology.orchestra.run.smithi193.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 2.4 MB/s | 44 kB 00:00 2023-06-23T01:17:43.782 INFO:teuthology.orchestra.run.smithi193.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 8.2 MB/s | 185 kB 00:00 2023-06-23T01:17:43.795 INFO:teuthology.orchestra.run.smithi193.stdout:(43/120): python3-asyncssh-2.7.0-2.el8.noarch.r 8.6 MB/s | 476 kB 00:00 2023-06-23T01:17:43.800 INFO:teuthology.orchestra.run.smithi193.stdout:(44/120): python3-cachetools-3.1.1-4.el8.noarch 1.8 MB/s | 33 kB 00:00 2023-06-23T01:17:43.812 INFO:teuthology.orchestra.run.smithi193.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 932 kB/s | 16 kB 00:00 2023-06-23T01:17:43.821 INFO:teuthology.orchestra.run.smithi193.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 8.1 MB/s | 173 kB 00:00 2023-06-23T01:17:43.836 INFO:teuthology.orchestra.run.smithi193.stdout:(47/120): python3-cherrypy-18.4.0-1.el8.noarch. 16 MB/s | 384 kB 00:00 2023-06-23T01:17:43.839 INFO:teuthology.orchestra.run.smithi193.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 4.3 MB/s | 81 kB 00:00 2023-06-23T01:17:43.853 INFO:teuthology.orchestra.run.smithi193.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 633 kB/s | 11 kB 00:00 2023-06-23T01:17:43.857 INFO:teuthology.orchestra.run.smithi193.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 999 kB/s | 18 kB 00:00 2023-06-23T01:17:43.882 INFO:teuthology.orchestra.run.smithi193.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 2.6 MB/s | 49 kB 00:00 2023-06-23T01:17:43.895 INFO:teuthology.orchestra.run.smithi193.stdout:(52/120): python3-kubernetes-11.0.0-6.el8.noarc 28 MB/s | 1.2 MB 00:00 2023-06-23T01:17:43.896 INFO:teuthology.orchestra.run.smithi193.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 2.8 MB/s | 59 kB 00:00 2023-06-23T01:17:43.914 INFO:teuthology.orchestra.run.smithi193.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 3.2 MB/s | 60 kB 00:00 2023-06-23T01:17:43.920 INFO:teuthology.orchestra.run.smithi193.stdout:(55/120): python3-pecan-1.3.2-9.el8.noarch.rpm 12 MB/s | 283 kB 00:00 2023-06-23T01:17:43.931 INFO:teuthology.orchestra.run.smithi193.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 890 kB/s | 16 kB 00:00 2023-06-23T01:17:43.938 INFO:teuthology.orchestra.run.smithi193.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.8 MB/s | 34 kB 00:00 2023-06-23T01:17:43.952 INFO:teuthology.orchestra.run.smithi193.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 9.7 MB/s | 196 kB 00:00 2023-06-23T01:17:43.957 INFO:teuthology.orchestra.run.smithi193.stdout:(59/120): python3-rsa-4.8-1.el8.noarch.rpm 3.3 MB/s | 61 kB 00:00 2023-06-23T01:17:43.968 INFO:teuthology.orchestra.run.smithi193.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 1.1 MB/s | 19 kB 00:00 2023-06-23T01:17:43.974 INFO:teuthology.orchestra.run.smithi193.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 1.3 MB/s | 24 kB 00:00 2023-06-23T01:17:43.985 INFO:teuthology.orchestra.run.smithi193.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 1.7 MB/s | 29 kB 00:00 2023-06-23T01:17:43.992 INFO:teuthology.orchestra.run.smithi193.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 1.5 MB/s | 27 kB 00:00 2023-06-23T01:17:44.003 INFO:teuthology.orchestra.run.smithi193.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 14 MB/s | 255 kB 00:00 2023-06-23T01:17:44.014 INFO:teuthology.orchestra.run.smithi193.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 12 MB/s | 251 kB 00:00 2023-06-23T01:17:44.020 INFO:teuthology.orchestra.run.smithi193.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 3.5 MB/s | 61 kB 00:00 2023-06-23T01:17:44.032 INFO:teuthology.orchestra.run.smithi193.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 4.5 MB/s | 85 kB 00:00 2023-06-23T01:17:44.037 INFO:teuthology.orchestra.run.smithi193.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 1.3 MB/s | 23 kB 00:00 2023-06-23T01:17:44.041 INFO:teuthology.orchestra.run.smithi193.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 19 MB/s | 70 kB 00:00 2023-06-23T01:17:44.093 INFO:teuthology.orchestra.run.smithi193.stdout:(70/120): thrift-0.13.0-2.el8.x86_64.rpm 29 MB/s | 1.7 MB 00:00 2023-06-23T01:17:44.135 INFO:teuthology.orchestra.run.smithi193.stdout:(71/120): python3-prettytable-0.7.2-14.el8.noar 1.0 MB/s | 44 kB 00:00 2023-06-23T01:17:44.160 INFO:teuthology.orchestra.run.smithi193.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.0 MB/s | 27 kB 00:00 2023-06-23T01:17:44.191 INFO:teuthology.orchestra.run.smithi193.stdout:(73/120): jq-1.5-12.el8.x86_64.rpm 5.2 MB/s | 161 kB 00:00 2023-06-23T01:17:44.222 INFO:teuthology.orchestra.run.smithi193.stdout:(74/120): flex-2.6.1-9.el8.x86_64.rpm 10 MB/s | 320 kB 00:00 2023-06-23T01:17:44.331 INFO:teuthology.orchestra.run.smithi193.stdout:(75/120): python3-scipy-1.0.0-20.module+el8.1.0 50 MB/s | 15 MB 00:00 2023-06-23T01:17:44.348 INFO:teuthology.orchestra.run.smithi193.stdout:(76/120): python3-werkzeug-0.12.2-4.el8.noarch. 3.6 MB/s | 457 kB 00:00 2023-06-23T01:17:44.376 INFO:teuthology.orchestra.run.smithi193.stdout:(77/120): python3-markupsafe-0.23-19.el8.x86_64 1.3 MB/s | 39 kB 00:00 2023-06-23T01:17:44.417 INFO:teuthology.orchestra.run.smithi193.stdout:(78/120): python3-babel-2.5.1-5.el8.noarch.rpm 56 MB/s | 4.8 MB 00:00 2023-06-23T01:17:44.426 INFO:teuthology.orchestra.run.smithi193.stdout:(79/120): bison-3.0.4-10.el8.x86_64.rpm 14 MB/s | 688 kB 00:00 2023-06-23T01:17:44.448 INFO:teuthology.orchestra.run.smithi193.stdout:(80/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 8.2 MB/s | 259 kB 00:00 2023-06-23T01:17:44.458 INFO:teuthology.orchestra.run.smithi193.stdout:(81/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 6.9 MB/s | 225 kB 00:00 2023-06-23T01:17:44.474 INFO:teuthology.orchestra.run.smithi193.stdout:(82/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 4.8 MB/s | 126 kB 00:00 2023-06-23T01:17:44.499 INFO:teuthology.orchestra.run.smithi193.stdout:(83/120): oniguruma-6.8.2-2.el8.x86_64.rpm 7.3 MB/s | 187 kB 00:00 2023-06-23T01:17:44.528 INFO:teuthology.orchestra.run.smithi193.stdout:(84/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 54 MB/s | 3.7 MB 00:00 2023-06-23T01:17:44.535 INFO:teuthology.orchestra.run.smithi193.stdout:(85/120): python3-jinja2-2.10.1-2.el8_0.noarch. 15 MB/s | 538 kB 00:00 2023-06-23T01:17:44.556 INFO:teuthology.orchestra.run.smithi193.stdout:(86/120): socat-1.7.3.3-2.el8.x86_64.rpm 11 MB/s | 302 kB 00:00 2023-06-23T01:17:44.563 INFO:teuthology.orchestra.run.smithi193.stdout:(87/120): python3-mako-1.0.6-13.el8.noarch.rpm 5.5 MB/s | 157 kB 00:00 2023-06-23T01:17:45.002 INFO:teuthology.orchestra.run.smithi193.stdout:(88/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 16 MB/s | 52 MB 00:03 2023-06-23T01:17:45.007 INFO:teuthology.orchestra.run.smithi193.stdout:(89/120): python3-pyasn1-modules-0.3.7-6.el8.no 244 kB/s | 110 kB 00:00 2023-06-23T01:17:45.011 INFO:teuthology.orchestra.run.smithi193.stdout:(90/120): librdkafka-0.11.4-1.el8.x86_64.rpm 789 kB/s | 353 kB 00:00 2023-06-23T01:17:45.043 INFO:teuthology.orchestra.run.smithi193.stdout:(91/120): libtool-2.4.6-25.el8.x86_64.rpm 19 MB/s | 709 kB 00:00 2023-06-23T01:17:45.049 INFO:teuthology.orchestra.run.smithi193.stdout:(92/120): python3-pytz-2017.2-9.el8.noarch.rpm 1.4 MB/s | 54 kB 00:00 2023-06-23T01:17:45.058 INFO:teuthology.orchestra.run.smithi193.stdout:(93/120): python3-jsonpointer-1.10-11.el8.noarc 363 kB/s | 20 kB 00:00 2023-06-23T01:17:45.095 INFO:teuthology.orchestra.run.smithi193.stdout:(94/120): automake-1.16.1-7.el8.noarch.rpm 19 MB/s | 713 kB 00:00 2023-06-23T01:17:45.118 INFO:teuthology.orchestra.run.smithi193.stdout:(95/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 835 kB/s | 19 kB 00:00 2023-06-23T01:17:45.147 INFO:teuthology.orchestra.run.smithi193.stdout:(96/120): openblas-threads-0.3.12-1.el8.x86_64. 46 MB/s | 4.7 MB 00:00 2023-06-23T01:17:45.148 INFO:teuthology.orchestra.run.smithi193.stdout:(97/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2023-06-23T01:17:45.163 INFO:teuthology.orchestra.run.smithi193.stdout:(98/120): openblas-0.3.12-1.el8.x86_64.rpm 40 MB/s | 4.6 MB 00:00 2023-06-23T01:17:45.175 INFO:teuthology.orchestra.run.smithi193.stdout:(99/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 8.7 MB/s | 220 kB 00:00 2023-06-23T01:17:45.176 INFO:teuthology.orchestra.run.smithi193.stdout:(100/120): python3-oauthlib-2.1.0-1.el8.noarch. 5.4 MB/s | 155 kB 00:00 2023-06-23T01:17:45.193 INFO:teuthology.orchestra.run.smithi193.stdout:(101/120): keyutils-libs-devel-1.5.10-6.el8.x86 1.6 MB/s | 48 kB 00:00 2023-06-23T01:17:45.200 INFO:teuthology.orchestra.run.smithi193.stdout:(102/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 8.2 MB/s | 229 kB 00:00 2023-06-23T01:17:45.201 INFO:teuthology.orchestra.run.smithi193.stdout:(103/120): python3-requests-oauthlib-1.0.0-1.el 1.6 MB/s | 43 kB 00:00 2023-06-23T01:17:45.225 INFO:teuthology.orchestra.run.smithi193.stdout:(104/120): libverto-devel-0.3.0-5.el8.x86_64.rp 741 kB/s | 18 kB 00:00 2023-06-23T01:17:45.235 INFO:teuthology.orchestra.run.smithi193.stdout:(105/120): pcre2-devel-10.32-2.el8.x86_64.rpm 17 MB/s | 605 kB 00:00 2023-06-23T01:17:45.249 INFO:teuthology.orchestra.run.smithi193.stdout:(106/120): openssl-devel-1.1.1g-15.el8_3.x86_64 42 MB/s | 2.3 MB 00:00 2023-06-23T01:17:45.250 INFO:teuthology.orchestra.run.smithi193.stdout:(107/120): fuse-2.9.7-12.el8.x86_64.rpm 3.2 MB/s | 83 kB 00:00 2023-06-23T01:17:45.259 INFO:teuthology.orchestra.run.smithi193.stdout:(108/120): elfutils-libelf-devel-0.182-3.el8.x8 2.4 MB/s | 59 kB 00:00 2023-06-23T01:17:45.276 INFO:teuthology.orchestra.run.smithi193.stdout:(109/120): libcom_err-devel-1.45.6-1.el8.x86_64 1.5 MB/s | 38 kB 00:00 2023-06-23T01:17:45.278 INFO:teuthology.orchestra.run.smithi193.stdout:(110/120): libquadmath-8.4.1-1.el8.x86_64.rpm 5.7 MB/s | 169 kB 00:00 2023-06-23T01:17:45.286 INFO:teuthology.orchestra.run.smithi193.stdout:(111/120): libsepol-devel-2.9-2.el8.x86_64.rpm 3.2 MB/s | 86 kB 00:00 2023-06-23T01:17:45.303 INFO:teuthology.orchestra.run.smithi193.stdout:(112/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.1 MB/s | 58 kB 00:00 2023-06-23T01:17:45.305 INFO:teuthology.orchestra.run.smithi193.stdout:(113/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.7 MB/s | 47 kB 00:00 2023-06-23T01:17:45.318 INFO:teuthology.orchestra.run.smithi193.stdout:(114/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 17 MB/s | 559 kB 00:00 2023-06-23T01:17:45.334 INFO:teuthology.orchestra.run.smithi193.stdout:(115/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 2.4 MB/s | 70 kB 00:00 2023-06-23T01:17:45.352 INFO:teuthology.orchestra.run.smithi193.stdout:(116/120): libgfortran-8.4.1-1.el8.x86_64.rpm 13 MB/s | 639 kB 00:00 2023-06-23T01:17:45.357 INFO:teuthology.orchestra.run.smithi193.stdout:(117/120): libselinux-devel-2.9-5.el8.x86_64.rp 5.1 MB/s | 200 kB 00:00 2023-06-23T01:17:45.360 INFO:teuthology.orchestra.run.smithi193.stdout:(118/120): libuuid-devel-2.32.1-27.el8.x86_64.r 3.7 MB/s | 97 kB 00:00 2023-06-23T01:17:45.596 INFO:teuthology.orchestra.run.smithi193.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 14 MB/s | 3.5 MB 00:00 2023-06-23T01:17:45.796 INFO:teuthology.orchestra.run.smithi193.stdout:(120/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 8.2 MB/s | 3.6 MB 00:00 2023-06-23T01:17:45.800 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:17:45.801 INFO:teuthology.orchestra.run.smithi193.stdout:Total 29 MB/s | 201 MB 00:06 2023-06-23T01:17:45.848 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2023-06-23T01:17:45.961 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2023-06-23T01:17:45.961 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2023-06-23T01:17:47.667 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2023-06-23T01:17:47.671 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2023-06-23T01:17:52.062 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2023-06-23T01:17:52.248 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T01:17:52.450 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-06-23T01:17:52.653 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T01:17:52.796 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T01:17:53.028 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T01:17:53.423 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T01:17:53.603 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:17:54.005 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:17:54.195 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:17:54.334 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:17:54.661 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:17:54.748 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:17:55.108 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:17:55.777 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T01:17:55.953 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:17:56.066 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:17:56.281 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T01:17:56.426 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T01:17:56.619 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:17:56.737 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:17:56.877 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:17:56.935 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:17:57.096 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-06-23T01:17:57.252 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T01:17:57.376 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T01:17:57.552 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T01:17:58.094 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T01:17:58.252 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T01:17:58.784 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T01:17:58.952 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T01:17:59.685 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T01:18:00.745 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-06-23T01:18:00.936 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-06-23T01:18:01.143 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-06-23T01:18:01.409 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-06-23T01:18:01.688 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-06-23T01:18:01.914 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T01:18:02.092 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T01:18:02.212 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T01:18:02.467 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:18:03.492 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-06-23T01:18:03.824 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-06-23T01:18:04.118 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-06-23T01:18:04.357 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-06-23T01:18:04.627 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-06-23T01:18:04.826 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T01:18:06.916 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T01:18:07.221 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-06-23T01:18:07.465 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T01:18:07.662 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T01:18:07.844 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T01:18:08.056 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T01:18:08.236 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T01:18:08.399 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-06-23T01:18:08.547 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-06-23T01:18:08.710 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-06-23T01:18:08.844 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-06-23T01:18:08.988 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-06-23T01:18:09.183 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-06-23T01:18:09.336 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-06-23T01:18:09.646 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-06-23T01:18:09.917 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-06-23T01:18:10.159 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-06-23T01:18:10.510 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-06-23T01:18:10.724 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-06-23T01:18:10.907 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-06-23T01:18:11.096 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-06-23T01:18:11.295 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-06-23T01:18:11.475 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T01:18:11.555 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T01:18:11.717 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-06-23T01:18:11.877 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-06-23T01:18:12.025 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T01:18:12.180 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T01:18:12.430 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T01:18:12.691 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T01:18:12.873 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T01:18:13.041 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T01:18:13.194 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T01:18:13.383 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T01:18:13.594 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T01:18:13.793 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T01:18:13.961 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T01:18:14.097 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T01:18:14.341 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T01:18:14.570 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T01:18:14.775 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T01:18:15.096 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T01:18:15.315 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T01:18:15.515 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T01:18:15.756 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T01:18:16.883 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T01:18:17.263 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T01:18:17.476 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T01:18:17.783 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T01:18:18.023 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T01:18:18.313 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T01:18:18.520 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T01:18:18.700 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T01:18:18.899 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T01:18:23.770 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:18:23.990 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:18:24.583 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:18:24.848 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:18:24.940 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:18:25.073 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:18:42.822 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:18:42.823 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2023-06-23T01:18:42.823 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2023-06-23T01:18:42.824 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /dev 2023-06-23T01:18:42.824 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /run 2023-06-23T01:18:42.824 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2023-06-23T01:18:42.824 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2023-06-23T01:18:42.824 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2023-06-23T01:18:42.824 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2023-06-23T01:18:42.824 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /dev 2023-06-23T01:18:42.825 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:18:43.153 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:18:44.636 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:18:44.795 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:18:45.017 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:18:45.153 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:18:45.543 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:18:45.686 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:18:45.703 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:18:48.353 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:18:48.489 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:18:48.684 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T01:18:48.872 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T01:18:49.027 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:18:49.243 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:18:49.419 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:18:50.389 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:18:50.560 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:18:50.660 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:18:51.024 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T01:18:51.194 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:18:51.231 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:18:51.401 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:18:51.413 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:18:53.004 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T01:18:53.227 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:18:53.318 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:18:53.443 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:18:53.486 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:18:53.618 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:18:53.966 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:18:54.135 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:18:59.066 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:18:59.376 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T01:18:59.550 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T01:19:00.254 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T01:19:00.596 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-06-23T01:19:00.808 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 111/122 2023-06-23T01:19:00.953 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-06-23T01:19:01.096 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T01:19:01.271 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T01:19:01.417 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T01:19:01.586 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T01:19:01.735 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T01:19:01.951 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-06-23T01:19:02.141 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T01:19:02.208 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T01:19:02.383 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T01:19:02.392 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T01:19:02.393 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:19:02.586 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:19:02.586 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:19:03.092 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:19:03.092 INFO:teuthology.orchestra.run.smithi193.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T01:19:03.092 INFO:teuthology.orchestra.run.smithi193.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T01:19:03.092 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:19:03.408 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T01:19:03.408 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T01:19:03.408 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T01:19:03.408 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:19:03.409 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T01:19:03.409 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:19:03.409 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:19:03.409 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:19:03.409 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T01:19:03.409 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:19:03.409 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T01:19:03.410 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T01:19:03.410 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:19:03.410 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T01:19:03.410 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T01:19:03.410 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T01:19:03.411 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T01:19:03.411 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T01:19:03.411 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T01:19:03.411 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T01:19:03.412 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T01:19:03.412 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T01:19:03.412 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T01:19:03.413 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T01:19:03.413 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T01:19:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T01:19:03.423 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T01:19:03.424 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:19:03.424 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T01:19:03.424 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T01:19:03.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T01:19:03.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T01:19:03.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T01:19:03.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T01:19:03.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T01:19:03.425 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T01:19:03.426 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T01:19:03.426 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T01:19:03.426 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T01:19:03.426 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T01:19:03.426 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T01:19:03.426 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T01:19:03.426 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T01:19:03.427 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T01:19:03.427 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T01:19:03.427 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T01:19:03.427 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T01:19:03.427 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T01:19:03.427 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T01:19:03.427 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T01:19:03.427 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T01:19:03.428 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T01:19:03.428 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T01:19:03.428 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T01:19:03.428 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T01:19:03.428 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T01:19:03.428 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T01:19:03.428 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T01:19:03.429 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T01:19:03.429 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T01:19:03.429 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T01:19:03.429 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T01:19:03.429 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T01:19:03.429 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T01:19:03.429 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T01:19:03.429 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T01:19:03.430 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T01:19:03.430 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T01:19:03.430 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T01:19:03.430 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T01:19:03.430 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T01:19:03.430 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-06-23T01:19:03.430 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-06-23T01:19:03.431 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-06-23T01:19:03.431 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-06-23T01:19:03.431 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-06-23T01:19:03.431 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-06-23T01:19:03.431 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-06-23T01:19:03.431 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-06-23T01:19:03.431 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-06-23T01:19:03.431 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-06-23T01:19:03.432 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-06-23T01:19:03.432 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-06-23T01:19:03.432 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-06-23T01:19:03.432 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-06-23T01:19:03.432 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-06-23T01:19:03.432 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-06-23T01:19:03.433 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-06-23T01:19:03.433 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-06-23T01:19:03.433 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-06-23T01:19:03.434 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-06-23T01:19:03.434 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-06-23T01:19:03.434 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-06-23T01:19:03.434 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-06-23T01:19:03.434 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-06-23T01:19:03.435 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-06-23T01:19:03.435 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-06-23T01:19:03.435 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-06-23T01:19:03.435 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-06-23T01:19:03.435 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-06-23T01:19:03.436 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-06-23T01:19:03.436 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-06-23T01:19:03.436 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-06-23T01:19:03.436 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-06-23T01:19:03.437 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-06-23T01:19:03.437 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-06-23T01:19:03.437 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-06-23T01:19:03.438 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-06-23T01:19:03.438 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-06-23T01:19:03.438 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-06-23T01:19:03.438 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-06-23T01:19:03.438 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-06-23T01:19:03.438 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-06-23T01:19:03.439 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-06-23T01:19:03.439 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-06-23T01:19:03.439 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-06-23T01:19:03.439 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-06-23T01:19:03.439 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-06-23T01:19:03.439 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T01:19:03.440 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T01:19:03.440 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T01:19:05.593 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:19:05.594 INFO:teuthology.orchestra.run.smithi193.stdout:Last metadata expiration check: 0:01:35 ago on Fri 23 Jun 2023 01:17:30 AM UTC. 2023-06-23T01:19:07.952 INFO:teuthology.orchestra.run.smithi193.stdout:Installed products updated. 2023-06-23T01:19:08.103 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:19:08.103 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2023-06-23T01:19:08.104 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.104 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.104 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2023-06-23T01:19:08.105 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T01:19:08.105 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T01:19:08.105 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.105 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.106 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.106 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.106 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.107 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.107 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.107 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.108 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.108 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.108 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.109 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.109 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.109 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.109 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.110 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.110 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.110 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.110 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.111 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.111 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.111 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.111 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:19:08.111 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T01:19:08.112 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T01:19:08.112 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T01:19:08.112 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T01:19:08.112 INFO:teuthology.orchestra.run.smithi193.stdout: jq-1.5-12.el8.x86_64 2023-06-23T01:19:08.112 INFO:teuthology.orchestra.run.smithi193.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T01:19:08.112 INFO:teuthology.orchestra.run.smithi193.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T01:19:08.113 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T01:19:08.113 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.113 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.113 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.113 INFO:teuthology.orchestra.run.smithi193.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T01:19:08.113 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T01:19:08.114 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T01:19:08.114 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T01:19:08.114 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T01:19:08.114 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.114 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.114 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T01:19:08.114 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.114 INFO:teuthology.orchestra.run.smithi193.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T01:19:08.115 INFO:teuthology.orchestra.run.smithi193.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T01:19:08.115 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T01:19:08.115 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T01:19:08.115 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T01:19:08.115 INFO:teuthology.orchestra.run.smithi193.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T01:19:08.115 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T01:19:08.115 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T01:19:08.116 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T01:19:08.116 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T01:19:08.116 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T01:19:08.116 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T01:19:08.116 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T01:19:08.116 INFO:teuthology.orchestra.run.smithi193.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T01:19:08.116 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T01:19:08.117 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T01:19:08.117 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T01:19:08.117 INFO:teuthology.orchestra.run.smithi193.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T01:19:08.117 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T01:19:08.117 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.117 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.117 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.117 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T01:19:08.118 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T01:19:08.118 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T01:19:08.118 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T01:19:08.118 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T01:19:08.118 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T01:19:08.119 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T01:19:08.119 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T01:19:08.119 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T01:19:08.119 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T01:19:08.120 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T01:19:08.120 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T01:19:08.120 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T01:19:08.121 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T01:19:08.121 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T01:19:08.121 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T01:19:08.121 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T01:19:08.122 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T01:19:08.122 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T01:19:08.122 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T01:19:08.122 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T01:19:08.123 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T01:19:08.123 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T01:19:08.123 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T01:19:08.124 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T01:19:08.124 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.124 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.124 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T01:19:08.124 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T01:19:08.125 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.125 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T01:19:08.125 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T01:19:08.125 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T01:19:08.125 INFO:teuthology.orchestra.run.smithi193.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T01:19:08.125 INFO:teuthology.orchestra.run.smithi193.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T01:19:08.125 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T01:19:08.125 INFO:teuthology.orchestra.run.smithi193.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T01:19:08.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T01:19:08.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T01:19:08.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T01:19:08.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T01:19:08.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T01:19:08.126 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T01:19:08.127 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.127 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.127 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:19:08.127 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T01:19:08.127 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T01:19:08.127 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T01:19:08.127 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T01:19:08.128 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T01:19:08.128 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:19:08.128 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2023-06-23T01:19:08.395 DEBUG:teuthology.parallel:result is None 2023-06-23T01:21:40.075 INFO:teuthology.orchestra.run.smithi067.stdout:Extra Packages for Enterprise Linux 61 kB/s | 16 MB 04:23 2023-06-23T01:21:43.477 INFO:teuthology.orchestra.run.smithi067.stdout:lab-extras 2.7 MB/s | 24 kB 00:00 2023-06-23T01:21:43.936 INFO:teuthology.orchestra.run.smithi067.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 103 MB/s | 38 MB 00:00 2023-06-23T01:21:49.767 INFO:teuthology.orchestra.run.smithi067.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 102 MB/s | 33 MB 00:00 2023-06-23T01:21:58.099 INFO:teuthology.orchestra.run.smithi067.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:21:58.100 INFO:teuthology.orchestra.run.smithi067.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:21:58.111 INFO:teuthology.orchestra.run.smithi067.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T01:21:58.112 INFO:teuthology.orchestra.run.smithi067.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T01:21:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T01:21:58.113 INFO:teuthology.orchestra.run.smithi067.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T01:21:58.115 INFO:teuthology.orchestra.run.smithi067.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T01:21:58.118 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T01:21:58.118 INFO:teuthology.orchestra.run.smithi067.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T01:21:58.645 INFO:teuthology.orchestra.run.smithi067.stdout:Dependencies resolved. 2023-06-23T01:21:58.657 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================================================== 2023-06-23T01:21:58.657 INFO:teuthology.orchestra.run.smithi067.stdout: Package Arch Version Repository Size 2023-06-23T01:21:58.658 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================================================== 2023-06-23T01:21:58.658 INFO:teuthology.orchestra.run.smithi067.stdout:Installing: 2023-06-23T01:21:58.658 INFO:teuthology.orchestra.run.smithi067.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T01:21:58.658 INFO:teuthology.orchestra.run.smithi067.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T01:21:58.659 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T01:21:58.659 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T01:21:58.659 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T01:21:58.660 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T01:21:58.660 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T01:21:58.660 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T01:21:58.661 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T01:21:58.661 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T01:21:58.661 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:21:58.661 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T01:21:58.662 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T01:21:58.662 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T01:21:58.662 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T01:21:58.662 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T01:21:58.663 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T01:21:58.663 INFO:teuthology.orchestra.run.smithi067.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T01:21:58.663 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T01:21:58.663 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T01:21:58.663 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T01:21:58.664 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T01:21:58.664 INFO:teuthology.orchestra.run.smithi067.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T01:21:58.664 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T01:21:58.665 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T01:21:58.665 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T01:21:58.665 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T01:21:58.665 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T01:21:58.665 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T01:21:58.666 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T01:21:58.666 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:21:58.666 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T01:21:58.666 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T01:21:58.666 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrading: 2023-06-23T01:21:58.666 INFO:teuthology.orchestra.run.smithi067.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:21:58.667 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T01:21:58.667 INFO:teuthology.orchestra.run.smithi067.stdout:Installing dependencies: 2023-06-23T01:21:58.667 INFO:teuthology.orchestra.run.smithi067.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T01:21:58.667 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T01:21:58.667 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T01:21:58.667 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T01:21:58.668 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T01:21:58.668 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T01:21:58.668 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:21:58.668 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T01:21:58.668 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T01:21:58.668 INFO:teuthology.orchestra.run.smithi067.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T01:21:58.669 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T01:21:58.669 INFO:teuthology.orchestra.run.smithi067.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T01:21:58.669 INFO:teuthology.orchestra.run.smithi067.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T01:21:58.669 INFO:teuthology.orchestra.run.smithi067.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T01:21:58.669 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T01:21:58.669 INFO:teuthology.orchestra.run.smithi067.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T01:21:58.669 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T01:21:58.670 INFO:teuthology.orchestra.run.smithi067.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T01:21:58.670 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T01:21:58.670 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T01:21:58.670 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T01:21:58.670 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T01:21:58.670 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T01:21:58.670 INFO:teuthology.orchestra.run.smithi067.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T01:21:58.671 INFO:teuthology.orchestra.run.smithi067.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T01:21:58.671 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T01:21:58.671 INFO:teuthology.orchestra.run.smithi067.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T01:21:58.671 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T01:21:58.672 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T01:21:58.672 INFO:teuthology.orchestra.run.smithi067.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T01:21:58.672 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T01:21:58.672 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T01:21:58.672 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T01:21:58.673 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T01:21:58.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T01:21:58.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T01:21:58.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T01:21:58.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T01:21:58.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T01:21:58.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T01:21:58.673 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T01:21:58.674 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T01:21:58.674 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T01:21:58.674 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T01:21:58.674 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T01:21:58.674 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T01:21:58.674 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T01:21:58.674 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T01:21:58.674 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T01:21:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T01:21:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:21:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T01:21:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T01:21:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T01:21:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T01:21:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T01:21:58.675 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T01:21:58.676 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T01:21:58.676 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T01:21:58.676 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T01:21:58.676 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T01:21:58.676 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T01:21:58.676 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T01:21:58.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T01:21:58.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T01:21:58.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T01:21:58.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T01:21:58.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:21:58.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T01:21:58.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T01:21:58.677 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T01:21:58.678 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T01:21:58.678 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T01:21:58.678 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T01:21:58.678 INFO:teuthology.orchestra.run.smithi067.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T01:21:58.678 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T01:21:58.678 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T01:21:58.678 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T01:21:58.678 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T01:21:58.679 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T01:21:58.679 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T01:21:58.679 INFO:teuthology.orchestra.run.smithi067.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T01:21:58.679 INFO:teuthology.orchestra.run.smithi067.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T01:21:58.679 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T01:21:58.679 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T01:21:58.679 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:21:58.680 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction Summary 2023-06-23T01:21:58.680 INFO:teuthology.orchestra.run.smithi067.stdout:====================================================================================================================== 2023-06-23T01:21:58.680 INFO:teuthology.orchestra.run.smithi067.stdout:Install 118 Packages 2023-06-23T01:21:58.680 INFO:teuthology.orchestra.run.smithi067.stdout:Upgrade 2 Packages 2023-06-23T01:21:58.680 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:21:58.681 INFO:teuthology.orchestra.run.smithi067.stdout:Total download size: 201 M 2023-06-23T01:21:58.681 INFO:teuthology.orchestra.run.smithi067.stdout:Downloading Packages: 2023-06-23T01:21:59.160 INFO:teuthology.orchestra.run.smithi067.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 42 kB/s | 6.4 kB 00:00 2023-06-23T01:21:59.561 INFO:teuthology.orchestra.run.smithi067.stdout:(2/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 9.3 MB/s | 5.2 MB 00:00 2023-06-23T01:21:59.604 INFO:teuthology.orchestra.run.smithi067.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 3.7 MB/s | 162 kB 00:00 2023-06-23T01:21:59.812 INFO:teuthology.orchestra.run.smithi067.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 11 MB/s | 2.3 MB 00:00 2023-06-23T01:21:59.814 INFO:teuthology.orchestra.run.smithi067.stdout:(5/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.3 MB/s | 873 kB 00:00 2023-06-23T01:21:59.955 INFO:teuthology.orchestra.run.smithi067.stdout:(6/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 11 MB/s | 1.6 MB 00:00 2023-06-23T01:22:01.023 INFO:teuthology.orchestra.run.smithi067.stdout:(7/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 12 MB/s | 23 MB 00:02 2023-06-23T01:22:01.147 INFO:teuthology.orchestra.run.smithi067.stdout:(8/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86_ 14 MB/s | 17 MB 00:01 2023-06-23T01:22:01.184 INFO:teuthology.orchestra.run.smithi067.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 664 kB/s | 24 kB 00:00 2023-06-23T01:22:02.669 INFO:teuthology.orchestra.run.smithi067.stdout:(10/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 10 MB/s | 17 MB 00:01 2023-06-23T01:22:02.701 INFO:teuthology.orchestra.run.smithi067.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 8.0 MB/s | 255 kB 00:00 2023-06-23T01:22:02.745 INFO:teuthology.orchestra.run.smithi067.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 565 kB/s | 25 kB 00:00 2023-06-23T01:22:02.863 INFO:teuthology.orchestra.run.smithi067.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 6.6 MB/s | 794 kB 00:00 2023-06-23T01:22:02.914 INFO:teuthology.orchestra.run.smithi067.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 3.3 MB/s | 174 kB 00:00 2023-06-23T01:22:02.966 INFO:teuthology.orchestra.run.smithi067.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 2.5 MB/s | 130 kB 00:00 2023-06-23T01:22:03.029 INFO:teuthology.orchestra.run.smithi067.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 7.8 MB/s | 506 kB 00:00 2023-06-23T01:22:03.059 INFO:teuthology.orchestra.run.smithi067.stdout:(17/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86 1.2 MB/s | 3.9 MB 00:03 2023-06-23T01:22:03.145 INFO:teuthology.orchestra.run.smithi067.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 519 kB/s | 45 kB 00:00 2023-06-23T01:22:03.312 INFO:teuthology.orchestra.run.smithi067.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 693 kB/s | 115 kB 00:00 2023-06-23T01:22:03.485 INFO:teuthology.orchestra.run.smithi067.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 1.1 MB/s | 190 kB 00:00 2023-06-23T01:22:03.594 INFO:teuthology.orchestra.run.smithi067.stdout:(21/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 9.8 MB/s | 5.5 MB 00:00 2023-06-23T01:22:03.656 INFO:teuthology.orchestra.run.smithi067.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 5.7 MB/s | 357 kB 00:00 2023-06-23T01:22:03.707 INFO:teuthology.orchestra.run.smithi067.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 2.0 MB/s | 107 kB 00:00 2023-06-23T01:22:03.754 INFO:teuthology.orchestra.run.smithi067.stdout:(24/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.9 MB/s | 91 kB 00:00 2023-06-23T01:22:03.776 INFO:teuthology.orchestra.run.smithi067.stdout:(25/120): python3-rados-17.2.6-719.g42ddc46e.el 1.2 MB/s | 371 kB 00:00 2023-06-23T01:22:04.284 INFO:teuthology.orchestra.run.smithi067.stdout:(26/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 17 MB/s | 52 MB 00:03 2023-06-23T01:22:04.286 INFO:teuthology.orchestra.run.smithi067.stdout:(27/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 339 kB/s | 173 kB 00:00 2023-06-23T01:22:04.310 INFO:teuthology.orchestra.run.smithi067.stdout:(28/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 6.2 MB/s | 3.5 MB 00:00 2023-06-23T01:22:04.311 INFO:teuthology.orchestra.run.smithi067.stdout:(29/120): ceph-grafana-dashboards-17.2.6-719.g4 879 kB/s | 23 kB 00:00 2023-06-23T01:22:04.377 INFO:teuthology.orchestra.run.smithi067.stdout:(30/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 1.3 MB/s | 121 kB 00:00 2023-06-23T01:22:04.449 INFO:teuthology.orchestra.run.smithi067.stdout:(31/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 12 MB/s | 1.7 MB 00:00 2023-06-23T01:22:04.493 INFO:teuthology.orchestra.run.smithi067.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 1.0 MB/s | 47 kB 00:00 2023-06-23T01:22:04.540 INFO:teuthology.orchestra.run.smithi067.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 307 kB/s | 14 kB 00:00 2023-06-23T01:22:04.575 INFO:teuthology.orchestra.run.smithi067.stdout:(34/120): ceph-mgr-modules-core-17.2.6-719.g42d 1.2 MB/s | 238 kB 00:00 2023-06-23T01:22:04.600 INFO:teuthology.orchestra.run.smithi067.stdout:(35/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 4.6 MB/s | 264 kB 00:00 2023-06-23T01:22:04.644 INFO:teuthology.orchestra.run.smithi067.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 380 kB/s | 18 kB 00:00 2023-06-23T01:22:04.651 INFO:teuthology.orchestra.run.smithi067.stdout:(37/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.0 MB/s | 80 kB 00:00 2023-06-23T01:22:04.660 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:22:04.683 INFO:teuthology.orchestra.run.smithi067.stdout:(38/120): ceph-mgr-diskprediction-local-17.2.6- 20 MB/s | 7.4 MB 00:00 2023-06-23T01:22:04.686 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:22:04.705 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:22:04.706 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:22:04.706 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:22:04.707 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:22:04.844 INFO:teuthology.orchestra.run.smithi067.stdout:(39/120): liboath-2.6.2-3.el8.x86_64.rpm 306 kB/s | 59 kB 00:00 2023-06-23T01:22:04.845 INFO:teuthology.orchestra.run.smithi067.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T01:22:04.855 INFO:teuthology.orchestra.run.smithi067.stdout:(40/120): libunwind-1.3.1-3.el8.x86_64.rpm 440 kB/s | 75 kB 00:00 2023-06-23T01:22:04.900 INFO:teuthology.orchestra.run.smithi067.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 988 kB/s | 44 kB 00:00 2023-06-23T01:22:04.921 INFO:teuthology.orchestra.run.smithi067.stdout:(42/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-06-23T01:22:04.952 INFO:teuthology.orchestra.run.smithi067.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.0 MB/s | 33 kB 00:00 2023-06-23T01:22:04.975 INFO:teuthology.orchestra.run.smithi067.stdout:(44/120): python3-asyncssh-2.7.0-2.el8.noarch.r 3.5 MB/s | 476 kB 00:00 2023-06-23T01:22:04.982 INFO:teuthology.orchestra.run.smithi067.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 547 kB/s | 16 kB 00:00 2023-06-23T01:22:04.989 INFO:teuthology.orchestra.run.smithi067.stdout:(46/120): python3-beautifulsoup4-4.6.3-2.el8.1. 2.1 MB/s | 185 kB 00:00 2023-06-23T01:22:05.012 INFO:teuthology.orchestra.run.smithi067.stdout:(47/120): python3-cheroot-8.5.2-1.el8.noarch.rp 4.6 MB/s | 173 kB 00:00 2023-06-23T01:22:05.032 INFO:teuthology.orchestra.run.smithi067.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 1.8 MB/s | 81 kB 00:00 2023-06-23T01:22:05.043 INFO:teuthology.orchestra.run.smithi067.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 372 kB/s | 11 kB 00:00 2023-06-23T01:22:05.044 INFO:teuthology.orchestra.run.smithi067.stdout:(50/120): python3-cherrypy-18.4.0-1.el8.noarch. 6.0 MB/s | 384 kB 00:00 2023-06-23T01:22:05.062 INFO:teuthology.orchestra.run.smithi067.stdout:(51/120): python3-jaraco-functools-2.0-4.el8.no 591 kB/s | 18 kB 00:00 2023-06-23T01:22:05.082 INFO:teuthology.orchestra.run.smithi067.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 1.3 MB/s | 49 kB 00:00 2023-06-23T01:22:05.097 INFO:teuthology.orchestra.run.smithi067.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 1.7 MB/s | 59 kB 00:00 2023-06-23T01:22:05.114 INFO:teuthology.orchestra.run.smithi067.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 1.9 MB/s | 60 kB 00:00 2023-06-23T01:22:05.126 INFO:teuthology.orchestra.run.smithi067.stdout:(55/120): python3-kubernetes-11.0.0-6.el8.noarc 14 MB/s | 1.2 MB 00:00 2023-06-23T01:22:05.145 INFO:teuthology.orchestra.run.smithi067.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 508 kB/s | 16 kB 00:00 2023-06-23T01:22:05.154 INFO:teuthology.orchestra.run.smithi067.stdout:(57/120): python3-pecan-1.3.2-9.el8.noarch.rpm 4.8 MB/s | 283 kB 00:00 2023-06-23T01:22:05.157 INFO:teuthology.orchestra.run.smithi067.stdout:(58/120): python3-repoze-lru-0.7-6.el8.noarch.r 1.1 MB/s | 34 kB 00:00 2023-06-23T01:22:05.188 INFO:teuthology.orchestra.run.smithi067.stdout:(59/120): python3-simplegeneric-0.8.1-17.el8.no 591 kB/s | 19 kB 00:00 2023-06-23T01:22:05.189 INFO:teuthology.orchestra.run.smithi067.stdout:(60/120): python3-routes-2.4.1-12.el8.noarch.rp 4.3 MB/s | 196 kB 00:00 2023-06-23T01:22:05.190 INFO:teuthology.orchestra.run.smithi067.stdout:(61/120): python3-rsa-4.8-1.el8.noarch.rpm 1.7 MB/s | 61 kB 00:00 2023-06-23T01:22:05.221 INFO:teuthology.orchestra.run.smithi067.stdout:(62/120): python3-singledispatch-3.4.0.3-18.el8 744 kB/s | 24 kB 00:00 2023-06-23T01:22:05.221 INFO:teuthology.orchestra.run.smithi067.stdout:(63/120): python3-tempora-1.14.1-5.el8.noarch.r 918 kB/s | 29 kB 00:00 2023-06-23T01:22:05.223 INFO:teuthology.orchestra.run.smithi067.stdout:(64/120): python3-trustme-0.6.0-4.el8.noarch.rp 863 kB/s | 27 kB 00:00 2023-06-23T01:22:05.259 INFO:teuthology.orchestra.run.smithi067.stdout:(65/120): python3-waitress-1.4.3-1.el8.noarch.r 6.5 MB/s | 255 kB 00:00 2023-06-23T01:22:05.272 INFO:teuthology.orchestra.run.smithi067.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-06-23T01:22:05.277 INFO:teuthology.orchestra.run.smithi067.stdout:(67/120): python3-webob-1.8.5-1.el8.1.noarch.rp 4.4 MB/s | 251 kB 00:00 2023-06-23T01:22:05.291 INFO:teuthology.orchestra.run.smithi067.stdout:(68/120): python3-webtest-2.0.33-1.el8.noarch.r 2.7 MB/s | 85 kB 00:00 2023-06-23T01:22:05.303 INFO:teuthology.orchestra.run.smithi067.stdout:(69/120): python3-zc-lockfile-2.0-2.el8.noarch. 755 kB/s | 23 kB 00:00 2023-06-23T01:22:05.304 INFO:teuthology.orchestra.run.smithi067.stdout:(70/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 5.0 MB/s | 70 kB 00:00 2023-06-23T01:22:05.338 INFO:teuthology.orchestra.run.smithi067.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.2 MB/s | 43 kB 00:00 2023-06-23T01:22:05.344 INFO:teuthology.orchestra.run.smithi067.stdout:(72/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 462 kB/s | 19 kB 00:00 2023-06-23T01:22:05.357 INFO:teuthology.orchestra.run.smithi067.stdout:(73/120): python3-oauthlib-2.1.0-1.el8.noarch.r 8.3 MB/s | 155 kB 00:00 2023-06-23T01:22:05.361 INFO:teuthology.orchestra.run.smithi067.stdout:(74/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 13 MB/s | 220 kB 00:00 2023-06-23T01:22:05.372 INFO:teuthology.orchestra.run.smithi067.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.2 MB/s | 48 kB 00:00 2023-06-23T01:22:05.381 INFO:teuthology.orchestra.run.smithi067.stdout:(76/120): thrift-0.13.0-2.el8.x86_64.rpm 17 MB/s | 1.7 MB 00:00 2023-06-23T01:22:05.385 INFO:teuthology.orchestra.run.smithi067.stdout:(77/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 9.4 MB/s | 229 kB 00:00 2023-06-23T01:22:05.389 INFO:teuthology.orchestra.run.smithi067.stdout:(78/120): python3-requests-oauthlib-1.0.0-1.el8 2.5 MB/s | 43 kB 00:00 2023-06-23T01:22:05.405 INFO:teuthology.orchestra.run.smithi067.stdout:(79/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 969 kB/s | 18 kB 00:00 2023-06-23T01:22:05.412 INFO:teuthology.orchestra.run.smithi067.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 26 MB/s | 605 kB 00:00 2023-06-23T01:22:05.420 INFO:teuthology.orchestra.run.smithi067.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 5.0 MB/s | 83 kB 00:00 2023-06-23T01:22:05.430 INFO:teuthology.orchestra.run.smithi067.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 3.3 MB/s | 59 kB 00:00 2023-06-23T01:22:05.435 INFO:teuthology.orchestra.run.smithi067.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 11 MB/s | 169 kB 00:00 2023-06-23T01:22:05.445 INFO:teuthology.orchestra.run.smithi067.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.7 MB/s | 38 kB 00:00 2023-06-23T01:22:05.452 INFO:teuthology.orchestra.run.smithi067.stdout:(85/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 33 MB/s | 2.3 MB 00:00 2023-06-23T01:22:05.454 INFO:teuthology.orchestra.run.smithi067.stdout:(86/120): libsepol-devel-2.9-2.el8.x86_64.rpm 4.7 MB/s | 86 kB 00:00 2023-06-23T01:22:05.460 INFO:teuthology.orchestra.run.smithi067.stdout:(87/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.7 MB/s | 58 kB 00:00 2023-06-23T01:22:05.466 INFO:teuthology.orchestra.run.smithi067.stdout:(88/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.4 MB/s | 47 kB 00:00 2023-06-23T01:22:05.478 INFO:teuthology.orchestra.run.smithi067.stdout:(89/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 23 MB/s | 559 kB 00:00 2023-06-23T01:22:05.480 INFO:teuthology.orchestra.run.smithi067.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 5.0 MB/s | 70 kB 00:00 2023-06-23T01:22:05.487 INFO:teuthology.orchestra.run.smithi067.stdout:(91/120): libgfortran-8.4.1-1.el8.x86_64.rpm 23 MB/s | 639 kB 00:00 2023-06-23T01:22:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 6.1 MB/s | 97 kB 00:00 2023-06-23T01:22:05.497 INFO:teuthology.orchestra.run.smithi067.stdout:(93/120): libselinux-devel-2.9-5.el8.x86_64.rpm 10 MB/s | 200 kB 00:00 2023-06-23T01:22:05.510 INFO:teuthology.orchestra.run.smithi067.stdout:(94/120): python3-prettytable-0.7.2-14.el8.noar 3.0 MB/s | 44 kB 00:00 2023-06-23T01:22:05.519 INFO:teuthology.orchestra.run.smithi067.stdout:(95/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.2 MB/s | 27 kB 00:00 2023-06-23T01:22:05.533 INFO:teuthology.orchestra.run.smithi067.stdout:(96/120): jq-1.5-12.el8.x86_64.rpm 6.9 MB/s | 161 kB 00:00 2023-06-23T01:22:05.538 INFO:teuthology.orchestra.run.smithi067.stdout:(97/120): flex-2.6.1-9.el8.x86_64.rpm 16 MB/s | 320 kB 00:00 2023-06-23T01:22:05.554 INFO:teuthology.orchestra.run.smithi067.stdout:(98/120): python3-werkzeug-0.12.2-4.el8.noarch. 21 MB/s | 457 kB 00:00 2023-06-23T01:22:05.568 INFO:teuthology.orchestra.run.smithi067.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 2.7 MB/s | 39 kB 00:00 2023-06-23T01:22:05.594 INFO:teuthology.orchestra.run.smithi067.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 26 MB/s | 688 kB 00:00 2023-06-23T01:22:05.613 INFO:teuthology.orchestra.run.smithi067.stdout:(101/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 14 MB/s | 259 kB 00:00 2023-06-23T01:22:05.638 INFO:teuthology.orchestra.run.smithi067.stdout:(102/120): python3-babel-2.5.1-5.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2023-06-23T01:22:05.642 INFO:teuthology.orchestra.run.smithi067.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 7.6 MB/s | 225 kB 00:00 2023-06-23T01:22:05.654 INFO:teuthology.orchestra.run.smithi067.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 7.7 MB/s | 126 kB 00:00 2023-06-23T01:22:05.675 INFO:teuthology.orchestra.run.smithi067.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 8.7 MB/s | 187 kB 00:00 2023-06-23T01:22:05.698 INFO:teuthology.orchestra.run.smithi067.stdout:(106/120): python3-jinja2-2.10.1-2.el8_0.noarch 23 MB/s | 538 kB 00:00 2023-06-23T01:22:05.816 INFO:teuthology.orchestra.run.smithi067.stdout:(107/120): python3-scipy-1.0.0-20.module+el8.1. 44 MB/s | 15 MB 00:00 2023-06-23T01:22:05.819 INFO:teuthology.orchestra.run.smithi067.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 2.4 MB/s | 302 kB 00:00 2023-06-23T01:22:05.839 INFO:teuthology.orchestra.run.smithi067.stdout:(109/120): python3-numpy-1.14.3-9.el8.x86_64.rp 20 MB/s | 3.7 MB 00:00 2023-06-23T01:22:05.839 INFO:teuthology.orchestra.run.smithi067.stdout:(110/120): python3-mako-1.0.6-13.el8.noarch.rpm 8.4 MB/s | 157 kB 00:00 2023-06-23T01:22:05.840 INFO:teuthology.orchestra.run.smithi067.stdout:(111/120): python3-pyasn1-modules-0.3.7-6.el8.n 5.5 MB/s | 110 kB 00:00 2023-06-23T01:22:05.852 INFO:teuthology.orchestra.run.smithi067.stdout:(112/120): librdkafka-0.11.4-1.el8.x86_64.rpm 17 MB/s | 353 kB 00:00 2023-06-23T01:22:05.853 INFO:teuthology.orchestra.run.smithi067.stdout:(113/120): python3-jsonpointer-1.10-11.el8.noar 1.0 MB/s | 20 kB 00:00 2023-06-23T01:22:05.867 INFO:teuthology.orchestra.run.smithi067.stdout:(114/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.8 MB/s | 54 kB 00:00 2023-06-23T01:22:05.871 INFO:teuthology.orchestra.run.smithi067.stdout:(115/120): libtool-2.4.6-25.el8.x86_64.rpm 21 MB/s | 709 kB 00:00 2023-06-23T01:22:05.901 INFO:teuthology.orchestra.run.smithi067.stdout:(116/120): automake-1.16.1-7.el8.noarch.rpm 24 MB/s | 713 kB 00:00 2023-06-23T01:22:05.946 INFO:teuthology.orchestra.run.smithi067.stdout:(117/120): openblas-threads-0.3.12-1.el8.x86_64 51 MB/s | 4.7 MB 00:00 2023-06-23T01:22:05.968 INFO:teuthology.orchestra.run.smithi067.stdout:(118/120): openblas-0.3.12-1.el8.x86_64.rpm 45 MB/s | 4.6 MB 00:00 2023-06-23T01:22:06.127 INFO:teuthology.orchestra.run.smithi067.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 15 MB/s | 3.5 MB 00:00 2023-06-23T01:22:06.679 INFO:teuthology.orchestra.run.smithi067.stdout:(120/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 4.9 MB/s | 3.6 MB 00:00 2023-06-23T01:22:06.689 INFO:teuthology.orchestra.run.smithi067.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:22:06.690 INFO:teuthology.orchestra.run.smithi067.stdout:Total 25 MB/s | 201 MB 00:08 2023-06-23T01:22:06.737 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction check 2023-06-23T01:22:06.863 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction check succeeded. 2023-06-23T01:22:06.863 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction test 2023-06-23T01:22:08.581 INFO:teuthology.orchestra.run.smithi067.stdout:Transaction test succeeded. 2023-06-23T01:22:08.585 INFO:teuthology.orchestra.run.smithi067.stdout:Running transaction 2023-06-23T01:22:13.490 INFO:teuthology.orchestra.run.smithi067.stdout: Preparing : 1/1 2023-06-23T01:22:13.701 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T01:22:13.929 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T01:22:14.072 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T01:22:14.812 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T01:22:15.210 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T01:22:15.545 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:22:15.950 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:22:16.153 INFO:teuthology.orchestra.run.smithi067.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:22:16.289 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:22:16.578 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:22:16.715 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:22:16.884 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:22:17.235 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-06-23T01:22:17.936 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T01:22:18.244 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:22:18.350 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:22:18.591 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T01:22:18.735 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T01:22:18.936 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:22:19.031 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:22:19.169 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:22:19.233 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:22:19.442 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-06-23T01:22:19.698 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-06-23T01:22:20.394 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-06-23T01:22:20.578 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-06-23T01:22:20.703 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T01:22:20.837 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T01:22:21.003 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T01:22:21.596 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T01:22:21.770 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T01:22:22.315 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T01:22:22.527 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T01:22:23.274 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T01:22:23.558 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-06-23T01:22:23.784 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-06-23T01:22:24.025 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T01:22:24.159 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T01:22:24.273 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T01:22:24.523 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:22:24.768 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-06-23T01:22:25.031 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-06-23T01:22:25.260 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T01:22:27.441 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T01:22:27.804 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-06-23T01:22:28.906 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-06-23T01:22:29.207 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-06-23T01:22:29.467 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-06-23T01:22:29.718 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T01:22:29.913 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T01:22:30.133 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T01:22:30.308 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T01:22:30.610 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T01:22:30.826 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-06-23T01:22:31.001 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-06-23T01:22:31.217 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-06-23T01:22:31.385 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-06-23T01:22:31.560 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T01:22:31.622 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T01:22:31.786 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-06-23T01:22:31.982 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-06-23T01:22:32.164 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-06-23T01:22:32.340 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-06-23T01:22:32.512 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-06-23T01:22:32.679 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-06-23T01:22:32.807 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-06-23T01:22:32.968 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-06-23T01:22:33.156 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-06-23T01:22:33.432 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-06-23T01:22:33.678 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-06-23T01:22:33.954 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-06-23T01:22:34.221 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-06-23T01:22:34.377 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T01:22:34.540 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T01:22:34.803 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T01:22:35.035 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T01:22:35.201 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T01:22:35.393 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T01:22:35.530 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T01:22:35.714 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T01:22:35.942 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T01:22:36.199 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T01:22:36.764 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T01:22:36.950 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T01:22:37.208 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T01:22:37.442 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T01:22:37.667 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T01:22:37.972 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T01:22:38.226 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T01:22:38.444 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T01:22:38.682 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T01:22:39.850 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T01:22:40.175 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T01:22:40.396 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T01:22:40.683 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T01:22:40.926 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T01:22:41.275 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T01:22:41.532 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T01:22:41.720 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T01:22:41.967 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T01:22:47.212 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:22:47.817 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:22:48.469 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:22:48.713 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:22:48.840 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:22:48.975 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:23:07.389 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:23:07.389 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /sys 2023-06-23T01:23:07.389 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /proc 2023-06-23T01:23:07.390 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /dev 2023-06-23T01:23:07.390 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /run 2023-06-23T01:23:07.390 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /mnt 2023-06-23T01:23:07.390 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /var/tmp 2023-06-23T01:23:07.390 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /home 2023-06-23T01:23:07.390 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /tmp 2023-06-23T01:23:07.390 INFO:teuthology.orchestra.run.smithi067.stdout:skipping the directory /dev 2023-06-23T01:23:07.391 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:07.772 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:23:09.356 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:23:09.502 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:23:09.737 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:23:09.872 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:23:10.286 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:23:10.472 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:23:10.490 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:23:13.453 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:23:13.563 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:23:13.736 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T01:23:13.953 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T01:23:14.112 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:23:14.339 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:23:14.567 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:23:15.600 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:23:15.813 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:23:15.919 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:23:16.324 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T01:23:16.518 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:23:16.557 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:23:16.704 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:23:16.716 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:23:18.372 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T01:23:18.568 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:23:18.646 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:23:18.821 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:23:18.866 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:23:19.012 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:23:19.382 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:23:19.587 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:23:24.909 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:23:25.928 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T01:23:26.267 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-06-23T01:23:26.468 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 110/122 2023-06-23T01:23:26.682 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-06-23T01:23:26.836 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T01:23:26.848 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T01:23:26.999 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T01:23:27.158 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T01:23:27.287 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T01:23:27.533 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T01:23:27.767 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T01:23:27.969 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T01:23:28.033 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T01:23:28.213 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T01:23:28.223 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T01:23:28.338 INFO:teuthology.orchestra.run.smithi067.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-06-23T01:23:28.339 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:23:28.548 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:23:28.549 INFO:teuthology.orchestra.run.smithi067.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:23:29.100 INFO:teuthology.orchestra.run.smithi067.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:23:29.101 INFO:teuthology.orchestra.run.smithi067.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T01:23:29.101 INFO:teuthology.orchestra.run.smithi067.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T01:23:29.101 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T01:23:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T01:23:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T01:23:29.523 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:23:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T01:23:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:23:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:23:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:23:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T01:23:29.524 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:23:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T01:23:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T01:23:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:23:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T01:23:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T01:23:29.525 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T01:23:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T01:23:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T01:23:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T01:23:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T01:23:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T01:23:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T01:23:29.526 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T01:23:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T01:23:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T01:23:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T01:23:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T01:23:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:23:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T01:23:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T01:23:29.527 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T01:23:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T01:23:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T01:23:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T01:23:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T01:23:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T01:23:29.528 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T01:23:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T01:23:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T01:23:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T01:23:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T01:23:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T01:23:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T01:23:29.529 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T01:23:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T01:23:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T01:23:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T01:23:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T01:23:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T01:23:29.530 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T01:23:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T01:23:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T01:23:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T01:23:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T01:23:29.531 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T01:23:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T01:23:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T01:23:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T01:23:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T01:23:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T01:23:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T01:23:29.532 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T01:23:29.533 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T01:23:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T01:23:29.534 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T01:23:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T01:23:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T01:23:29.535 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T01:23:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T01:23:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T01:23:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T01:23:29.536 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-06-23T01:23:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-06-23T01:23:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-06-23T01:23:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-06-23T01:23:29.537 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-06-23T01:23:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-06-23T01:23:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-06-23T01:23:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-06-23T01:23:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-06-23T01:23:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-06-23T01:23:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-06-23T01:23:29.538 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-06-23T01:23:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-06-23T01:23:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-06-23T01:23:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-06-23T01:23:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-06-23T01:23:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-06-23T01:23:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-06-23T01:23:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-06-23T01:23:29.539 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-06-23T01:23:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-06-23T01:23:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-06-23T01:23:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-06-23T01:23:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-06-23T01:23:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-06-23T01:23:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-06-23T01:23:29.540 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-06-23T01:23:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-06-23T01:23:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-06-23T01:23:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-06-23T01:23:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-06-23T01:23:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-06-23T01:23:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-06-23T01:23:29.541 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-06-23T01:23:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-06-23T01:23:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-06-23T01:23:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-06-23T01:23:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-06-23T01:23:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-06-23T01:23:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-06-23T01:23:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-06-23T01:23:29.542 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-06-23T01:23:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-06-23T01:23:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-06-23T01:23:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-06-23T01:23:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-06-23T01:23:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-06-23T01:23:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T01:23:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T01:23:29.543 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T01:23:31.708 INFO:teuthology.orchestra.run.smithi067.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:23:31.709 INFO:teuthology.orchestra.run.smithi067.stdout:Last metadata expiration check: 0:01:42 ago on Fri 23 Jun 2023 01:21:49 AM UTC. 2023-06-23T01:23:33.941 INFO:teuthology.orchestra.run.smithi067.stdout:Installed products updated. 2023-06-23T01:23:34.104 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:34.105 INFO:teuthology.orchestra.run.smithi067.stdout:Upgraded: 2023-06-23T01:23:34.105 INFO:teuthology.orchestra.run.smithi067.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.106 INFO:teuthology.orchestra.run.smithi067.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.106 INFO:teuthology.orchestra.run.smithi067.stdout:Installed: 2023-06-23T01:23:34.106 INFO:teuthology.orchestra.run.smithi067.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T01:23:34.106 INFO:teuthology.orchestra.run.smithi067.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T01:23:34.107 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.107 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.107 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.108 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.108 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.108 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.109 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.109 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.109 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.110 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.110 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.110 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.111 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.111 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.111 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.112 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.112 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.112 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.112 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.113 INFO:teuthology.orchestra.run.smithi067.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.113 INFO:teuthology.orchestra.run.smithi067.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.113 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.114 INFO:teuthology.orchestra.run.smithi067.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:23:34.114 INFO:teuthology.orchestra.run.smithi067.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T01:23:34.114 INFO:teuthology.orchestra.run.smithi067.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T01:23:34.114 INFO:teuthology.orchestra.run.smithi067.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T01:23:34.115 INFO:teuthology.orchestra.run.smithi067.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T01:23:34.115 INFO:teuthology.orchestra.run.smithi067.stdout: jq-1.5-12.el8.x86_64 2023-06-23T01:23:34.115 INFO:teuthology.orchestra.run.smithi067.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T01:23:34.115 INFO:teuthology.orchestra.run.smithi067.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T01:23:34.116 INFO:teuthology.orchestra.run.smithi067.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T01:23:34.116 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.116 INFO:teuthology.orchestra.run.smithi067.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.117 INFO:teuthology.orchestra.run.smithi067.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.117 INFO:teuthology.orchestra.run.smithi067.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T01:23:34.117 INFO:teuthology.orchestra.run.smithi067.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T01:23:34.117 INFO:teuthology.orchestra.run.smithi067.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T01:23:34.117 INFO:teuthology.orchestra.run.smithi067.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T01:23:34.118 INFO:teuthology.orchestra.run.smithi067.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T01:23:34.118 INFO:teuthology.orchestra.run.smithi067.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.118 INFO:teuthology.orchestra.run.smithi067.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.118 INFO:teuthology.orchestra.run.smithi067.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T01:23:34.118 INFO:teuthology.orchestra.run.smithi067.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.118 INFO:teuthology.orchestra.run.smithi067.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T01:23:34.118 INFO:teuthology.orchestra.run.smithi067.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T01:23:34.119 INFO:teuthology.orchestra.run.smithi067.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T01:23:34.119 INFO:teuthology.orchestra.run.smithi067.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T01:23:34.119 INFO:teuthology.orchestra.run.smithi067.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T01:23:34.119 INFO:teuthology.orchestra.run.smithi067.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T01:23:34.119 INFO:teuthology.orchestra.run.smithi067.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T01:23:34.119 INFO:teuthology.orchestra.run.smithi067.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T01:23:34.119 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T01:23:34.120 INFO:teuthology.orchestra.run.smithi067.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T01:23:34.120 INFO:teuthology.orchestra.run.smithi067.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T01:23:34.120 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T01:23:34.120 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T01:23:34.120 INFO:teuthology.orchestra.run.smithi067.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T01:23:34.120 INFO:teuthology.orchestra.run.smithi067.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T01:23:34.120 INFO:teuthology.orchestra.run.smithi067.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T01:23:34.121 INFO:teuthology.orchestra.run.smithi067.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T01:23:34.121 INFO:teuthology.orchestra.run.smithi067.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T01:23:34.121 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T01:23:34.121 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.121 INFO:teuthology.orchestra.run.smithi067.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.121 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.121 INFO:teuthology.orchestra.run.smithi067.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T01:23:34.122 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T01:23:34.122 INFO:teuthology.orchestra.run.smithi067.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T01:23:34.122 INFO:teuthology.orchestra.run.smithi067.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T01:23:34.122 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T01:23:34.122 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T01:23:34.122 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T01:23:34.122 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T01:23:34.123 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T01:23:34.123 INFO:teuthology.orchestra.run.smithi067.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T01:23:34.123 INFO:teuthology.orchestra.run.smithi067.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T01:23:34.123 INFO:teuthology.orchestra.run.smithi067.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T01:23:34.123 INFO:teuthology.orchestra.run.smithi067.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T01:23:34.123 INFO:teuthology.orchestra.run.smithi067.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T01:23:34.123 INFO:teuthology.orchestra.run.smithi067.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T01:23:34.124 INFO:teuthology.orchestra.run.smithi067.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T01:23:34.124 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T01:23:34.124 INFO:teuthology.orchestra.run.smithi067.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T01:23:34.124 INFO:teuthology.orchestra.run.smithi067.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T01:23:34.124 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T01:23:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T01:23:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T01:23:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T01:23:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T01:23:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T01:23:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.125 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T01:23:34.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T01:23:34.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T01:23:34.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T01:23:34.126 INFO:teuthology.orchestra.run.smithi067.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T01:23:34.127 INFO:teuthology.orchestra.run.smithi067.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T01:23:34.127 INFO:teuthology.orchestra.run.smithi067.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T01:23:34.127 INFO:teuthology.orchestra.run.smithi067.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T01:23:34.127 INFO:teuthology.orchestra.run.smithi067.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T01:23:34.128 INFO:teuthology.orchestra.run.smithi067.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T01:23:34.129 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T01:23:34.129 INFO:teuthology.orchestra.run.smithi067.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T01:23:34.130 INFO:teuthology.orchestra.run.smithi067.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T01:23:34.130 INFO:teuthology.orchestra.run.smithi067.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T01:23:34.130 INFO:teuthology.orchestra.run.smithi067.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T01:23:34.130 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.131 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.131 INFO:teuthology.orchestra.run.smithi067.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:23:34.131 INFO:teuthology.orchestra.run.smithi067.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T01:23:34.132 INFO:teuthology.orchestra.run.smithi067.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T01:23:34.132 INFO:teuthology.orchestra.run.smithi067.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T01:23:34.132 INFO:teuthology.orchestra.run.smithi067.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T01:23:34.133 INFO:teuthology.orchestra.run.smithi067.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T01:23:34.133 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:34.134 INFO:teuthology.orchestra.run.smithi067.stdout:Complete! 2023-06-23T01:23:34.385 DEBUG:teuthology.parallel:result is None 2023-06-23T01:23:34.386 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:23:34.609 DEBUG:teuthology.orchestra.run.smithi067:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T01:23:34.633 INFO:teuthology.orchestra.run.smithi067.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T01:23:34.634 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T01:23:34.635 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T01:23:34.636 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:23:34.775 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T01:23:34.801 INFO:teuthology.orchestra.run.smithi193.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T01:23:34.801 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T01:23:34.801 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T01:23:34.802 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-23T01:23:34.802 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:34.803 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T01:23:34.846 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:34.846 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T01:23:34.891 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-23T01:23:34.891 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:34.892 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T01:23:34.917 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T01:23:34.985 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:34.985 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T01:23:35.013 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T01:23:35.080 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-23T01:23:35.081 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:35.081 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T01:23:35.108 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T01:23:35.174 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:35.174 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T01:23:35.201 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T01:23:35.266 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-23T01:23:35.267 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:35.267 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T01:23:35.292 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T01:23:35.361 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:35.361 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T01:23:35.387 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T01:23:35.452 INFO:teuthology.run_tasks:Running task ceph... 2023-06-23T01:23:35.559 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-06-23T01:23:35.560 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 777 /var/log/ceph 2023-06-23T01:23:35.562 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /var/log/ceph 2023-06-23T01:23:35.587 INFO:tasks.ceph:Disabling ceph logrotate... 2023-06-23T01:23:35.588 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T01:23:35.629 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T01:23:35.653 INFO:tasks.ceph:Creating extra log directories... 2023-06-23T01:23:35.654 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T01:23:35.701 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T01:23:35.733 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-06-23T01:23:35.734 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-06-23T01:23:35.734 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-06-23T01:23:35.734 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:35.734 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T01:23:35.780 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T01:23:35.897 DEBUG:teuthology.orchestra.run.smithi067:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T01:23:35.924 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:35.925 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T01:23:35.941 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T01:23:36.053 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T01:23:36.083 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-06-23T01:23:36.084 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'client.0': {'debug client': 20, 'debug objectcacher': 20, 'debug objecter': 20}, 'client.1': {'debug client': 20, 'debug objectcacher': 20, 'debug objecter': 20}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd backfillfull ratio': 0.6, 'mon osd full ratio': 0.7, 'mon osd nearfull ratio': 0.6}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'memstore device bytes': 200000000, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 1.0, 'osd mon report interval': 5, 'osd objectstore': 'memstore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL', 'PG_RECOVERY_FULL', 'PG_DEGRADED'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-06-23T01:23:36.085 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945', 'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7311945', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy'}, 'ceph': {'cephfs': {'ec_profile': ['disabled']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'client.0': {'debug client': 20, 'debug objectcacher': 20, 'debug objecter': 20}, 'client.1': {'debug client': 20, 'debug objectcacher': 20, 'debug objecter': 20}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120, 'mon osd backfillfull ratio': 0.6, 'mon osd full ratio': 0.7, 'mon osd nearfull ratio': 0.6}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'memstore device bytes': 200000000, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 1.0, 'osd mon report interval': 5, 'osd objectstore': 'memstore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL', 'PG_RECOVERY_FULL', 'PG_DEGRADED'], 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9037, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'targets': {'smithi067.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSI9ftZPlRt+IDHUi5hLfCxz7QLJGWQzw8RXkHRFcGsdhLvRaI9dDGsCupQX8AnPEPQNjqvtNkeAzBBPaJrU6mukCrWlcP5De7lGHelE7UA8J1IvH79bsdvgSWpjp3ed5NZVF6A0urzpmxol78o4MUZMibWHiHB+cHw+i18aYXMwKI5H6NkeiqGmD8y0PH5B8rM6vYJg5jVctgUIg/mM+1GV2ZsRKuIQmdiE+6VmHN8aTg4PLF/DWBX2fB2O4j867QZnZUyKaboSKtJgCf8V7FOvYSlewnDvr+yDEwVeXfLxgHR6dLxn+Vl/erM1IIFyqtjJHXASH0J1rOCqENj2g+N5fHaMGGmyTBtjza0h5lVBUSrxkg7BwqzR6xuo/3yoj88cDBrvdcq6f8w6SNcngV35J0dh1x8vERPrjjSn+p10IKVMKkDreofHWSNGsA9CxskVD2dB+40gSsGzQs/ngcwtihMwarIdnHHSAgve3Xt9+vSqrroi44BLzgpooNQss=', 'smithi193.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBED64u4gjLzpTDT6XA3p9m/odRaKWL8FincvF3djpWK1PElxZKPfNAr+gOw0bHvTOWBCdhDdQlZfrX7eCSWhduQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_full']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '076bbebc42a14f7d568aaa78eabb0038327bcb23', 'timestamp': '2023-06-22_15:25:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-06-23T01:23:36.086 DEBUG:teuthology.orchestra.run.smithi067:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T01:23:36.089 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T01:23:36.133 DEBUG:teuthology.orchestra.run.smithi067:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T01:23:36.144 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T01:23:36.197 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:36.198 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:23:36.223 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:23:36.223 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_1 2023-06-23T01:23:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:23:36.278 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:23:36.278 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 212385 Links: 1 2023-06-23T01:23:36.278 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:23:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:23:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-06-23 01:23:29.008054722 +0000 2023-06-23T01:23:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-06-23 01:16:40.186811372 +0000 2023-06-23T01:23:36.279 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-06-23 01:16:40.186811372 +0000 2023-06-23T01:23:36.279 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2023-06-23T01:23:36.279 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:23:36.341 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-06-23T01:23:36.342 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-06-23T01:23:36.342 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000236551 s, 2.2 MB/s 2023-06-23T01:23:36.343 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:23:36.400 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_2 2023-06-23T01:23:36.455 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:23:36.456 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:23:36.456 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 207785 Links: 1 2023-06-23T01:23:36.456 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:23:36.456 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:23:36.456 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-06-23 01:23:29.008054722 +0000 2023-06-23T01:23:36.457 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-06-23 01:16:40.729799361 +0000 2023-06-23T01:23:36.457 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-06-23 01:16:40.729799361 +0000 2023-06-23T01:23:36.457 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2023-06-23T01:23:36.457 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:23:36.521 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-06-23T01:23:36.522 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-06-23T01:23:36.522 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000227252 s, 2.3 MB/s 2023-06-23T01:23:36.523 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:23:36.579 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_3 2023-06-23T01:23:36.634 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:23:36.635 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:23:36.635 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 215047 Links: 1 2023-06-23T01:23:36.635 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:23:36.636 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:23:36.636 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-06-23 01:23:29.008054722 +0000 2023-06-23T01:23:36.636 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-06-23 01:16:41.238788102 +0000 2023-06-23T01:23:36.636 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-06-23 01:16:41.238788102 +0000 2023-06-23T01:23:36.636 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2023-06-23T01:23:36.637 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:23:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-06-23T01:23:36.701 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-06-23T01:23:36.702 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000203329 s, 2.5 MB/s 2023-06-23T01:23:36.703 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:23:36.760 DEBUG:teuthology.orchestra.run.smithi067:> stat /dev/vg_nvme/lv_4 2023-06-23T01:23:36.815 INFO:teuthology.orchestra.run.smithi067.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:23:36.815 INFO:teuthology.orchestra.run.smithi067.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:23:36.816 INFO:teuthology.orchestra.run.smithi067.stdout:Device: 6h/6d Inode: 214476 Links: 1 2023-06-23T01:23:36.816 INFO:teuthology.orchestra.run.smithi067.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:23:36.816 INFO:teuthology.orchestra.run.smithi067.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:23:36.816 INFO:teuthology.orchestra.run.smithi067.stdout:Access: 2023-06-23 01:23:29.008054722 +0000 2023-06-23T01:23:36.816 INFO:teuthology.orchestra.run.smithi067.stdout:Modify: 2023-06-23 01:16:41.756776644 +0000 2023-06-23T01:23:36.816 INFO:teuthology.orchestra.run.smithi067.stdout:Change: 2023-06-23 01:16:41.756776644 +0000 2023-06-23T01:23:36.816 INFO:teuthology.orchestra.run.smithi067.stdout: Birth: - 2023-06-23T01:23:36.817 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:23:36.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records in 2023-06-23T01:23:36.879 INFO:teuthology.orchestra.run.smithi067.stderr:1+0 records out 2023-06-23T01:23:36.879 INFO:teuthology.orchestra.run.smithi067.stderr:512 bytes copied, 0.000198994 s, 2.6 MB/s 2023-06-23T01:23:36.880 DEBUG:teuthology.orchestra.run.smithi067:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:23:36.936 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:36.936 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:36.937 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:23:36.953 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:23:36.953 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2023-06-23T01:23:37.009 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:23:37.010 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:23:37.010 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 215057 Links: 1 2023-06-23T01:23:37.010 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:23:37.011 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:23:37.011 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-06-23 01:19:03.005879444 +0000 2023-06-23T01:23:37.011 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-06-23 01:14:21.150058977 +0000 2023-06-23T01:23:37.011 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-06-23 01:14:21.150058977 +0000 2023-06-23T01:23:37.012 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-06-23T01:23:37.012 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:23:37.073 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-06-23T01:23:37.074 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-06-23T01:23:37.074 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.00027545 s, 1.9 MB/s 2023-06-23T01:23:37.075 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:23:37.131 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2023-06-23T01:23:37.186 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:23:37.187 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:23:37.187 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 210533 Links: 1 2023-06-23T01:23:37.187 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:23:37.187 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:23:37.187 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-06-23 01:19:03.005879444 +0000 2023-06-23T01:23:37.187 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-06-23 01:14:21.650051063 +0000 2023-06-23T01:23:37.187 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-06-23 01:14:21.650051063 +0000 2023-06-23T01:23:37.188 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-06-23T01:23:37.188 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:23:37.249 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-06-23T01:23:37.249 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-06-23T01:23:37.249 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000224771 s, 2.3 MB/s 2023-06-23T01:23:37.250 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:23:37.306 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2023-06-23T01:23:37.361 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:23:37.361 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:23:37.361 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 212481 Links: 1 2023-06-23T01:23:37.361 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:23:37.361 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:23:37.362 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-06-23 01:19:03.005879444 +0000 2023-06-23T01:23:37.362 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-06-23 01:14:22.169042848 +0000 2023-06-23T01:23:37.362 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-06-23 01:14:22.169042848 +0000 2023-06-23T01:23:37.362 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-06-23T01:23:37.362 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:23:37.424 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-06-23T01:23:37.424 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-06-23T01:23:37.424 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000220055 s, 2.3 MB/s 2023-06-23T01:23:37.425 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:23:37.480 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2023-06-23T01:23:37.535 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:23:37.535 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:23:37.535 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 6h/6d Inode: 210653 Links: 1 2023-06-23T01:23:37.535 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:23:37.536 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:23:37.536 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2023-06-23 01:19:03.005879444 +0000 2023-06-23T01:23:37.536 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2023-06-23 01:14:22.668034949 +0000 2023-06-23T01:23:37.536 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2023-06-23 01:14:22.668034949 +0000 2023-06-23T01:23:37.536 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: - 2023-06-23T01:23:37.537 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:23:37.598 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2023-06-23T01:23:37.598 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2023-06-23T01:23:37.598 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000200336 s, 2.6 MB/s 2023-06-23T01:23:37.599 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:23:37.654 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:37.655 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T01:23:37.655 INFO:tasks.ceph:Generating config... 2023-06-23T01:23:37.656 INFO:tasks.ceph:[client] client mount timeout = 600 2023-06-23T01:23:37.657 INFO:tasks.ceph:[client] debug client = 20 2023-06-23T01:23:37.657 INFO:tasks.ceph:[client] debug ms = 1 2023-06-23T01:23:37.657 INFO:tasks.ceph:[client] pid file = 2023-06-23T01:23:37.657 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-06-23T01:23:37.658 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-06-23T01:23:37.658 INFO:tasks.ceph:[client.0] debug client = 20 2023-06-23T01:23:37.658 INFO:tasks.ceph:[client.0] debug objectcacher = 20 2023-06-23T01:23:37.658 INFO:tasks.ceph:[client.0] debug objecter = 20 2023-06-23T01:23:37.658 INFO:tasks.ceph:[client.1] debug client = 20 2023-06-23T01:23:37.659 INFO:tasks.ceph:[client.1] debug objectcacher = 20 2023-06-23T01:23:37.659 INFO:tasks.ceph:[client.1] debug objecter = 20 2023-06-23T01:23:37.659 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-06-23T01:23:37.659 INFO:tasks.ceph:[mds] debug mds = 20 2023-06-23T01:23:37.659 INFO:tasks.ceph:[mds] debug ms = 1 2023-06-23T01:23:37.660 INFO:tasks.ceph:[mds] mds debug frag = True 2023-06-23T01:23:37.660 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-06-23T01:23:37.661 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-06-23T01:23:37.661 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-06-23T01:23:37.661 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-06-23T01:23:37.661 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-06-23T01:23:37.661 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-06-23T01:23:37.662 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-06-23T01:23:37.662 INFO:tasks.ceph:[mgr] debug ms = 1 2023-06-23T01:23:37.662 INFO:tasks.ceph:[mon] debug mon = 20 2023-06-23T01:23:37.662 INFO:tasks.ceph:[mon] debug ms = 1 2023-06-23T01:23:37.662 INFO:tasks.ceph:[mon] debug paxos = 20 2023-06-23T01:23:37.663 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-06-23T01:23:37.663 INFO:tasks.ceph:[mon] mon osd backfillfull ratio = 0.6 2023-06-23T01:23:37.663 INFO:tasks.ceph:[mon] mon osd full ratio = 0.7 2023-06-23T01:23:37.663 INFO:tasks.ceph:[mon] mon osd nearfull ratio = 0.6 2023-06-23T01:23:37.664 INFO:tasks.ceph:[osd] bdev async discard = True 2023-06-23T01:23:37.664 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-06-23T01:23:37.665 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-06-23T01:23:37.665 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-06-23T01:23:37.665 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-06-23T01:23:37.665 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-06-23T01:23:37.665 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-06-23T01:23:37.666 INFO:tasks.ceph:[osd] debug ms = 1 2023-06-23T01:23:37.666 INFO:tasks.ceph:[osd] debug osd = 20 2023-06-23T01:23:37.666 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-06-23T01:23:37.666 INFO:tasks.ceph:[osd] memstore device bytes = 200000000 2023-06-23T01:23:37.666 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-06-23T01:23:37.667 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-06-23T01:23:37.667 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-06-23T01:23:37.667 INFO:tasks.ceph:[osd] osd failsafe full ratio = 1.0 2023-06-23T01:23:37.667 INFO:tasks.ceph:[osd] osd mon report interval = 5 2023-06-23T01:23:37.667 INFO:tasks.ceph:[osd] osd objectstore = memstore 2023-06-23T01:23:37.667 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-06-23T01:23:37.668 INFO:tasks.ceph:Setting up mon.a... 2023-06-23T01:23:37.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-06-23T01:23:37.720 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.keyring 2023-06-23T01:23:37.722 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-06-23T01:23:37.763 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:23:37.829 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.67'), ('mon.b', '172.21.15.193'), ('mon.c', '[v2:172.21.15.193:3301,v1:172.21.15.193:6790]')] 2023-06-23T01:23:37.830 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.67,172.21.15.193,[v2:172.21.15.193:3301,v1:172.21.15.193:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'memstore device bytes': 200000000, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 1.0, 'osd mon report interval': 5, 'osd objectstore': 'memstore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120, 'mon osd backfillfull ratio': 0.6, 'mon osd full ratio': 0.7, 'mon osd nearfull ratio': 0.6}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'client.0': {'debug client': 20, 'debug objectcacher': 20, 'debug objecter': 20}, 'client.1': {'debug client': 20, 'debug objectcacher': 20, 'debug objecter': 20}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-06-23T01:23:37.832 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:37.832 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T01:23:37.885 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.67 --add b 172.21.15.193 --addv c '[v2:172.21.15.193:3301,v1:172.21.15.193:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:37.969 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: generated fsid 0068cd93-d7dd-4502-a85d-c8368ea7e9d2 2023-06-23T01:23:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:setting min_mon_release = octopus 2023-06-23T01:23:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:epoch 0 2023-06-23T01:23:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:fsid 0068cd93-d7dd-4502-a85d-c8368ea7e9d2 2023-06-23T01:23:37.970 INFO:teuthology.orchestra.run.smithi067.stdout:last_changed 2023-06-23T01:23:37.968048+0000 2023-06-23T01:23:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:created 2023-06-23T01:23:37.968048+0000 2023-06-23T01:23:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:min_mon_release 15 (octopus) 2023-06-23T01:23:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:election_strategy: 1 2023-06-23T01:23:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:0: [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] mon.a 2023-06-23T01:23:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:1: [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon.b 2023-06-23T01:23:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:2: [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] mon.c 2023-06-23T01:23:37.971 INFO:teuthology.orchestra.run.smithi067.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-06-23T01:23:37.973 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T01:23:38.029 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 0068cd93-d7dd-4502-a85d-c8368ea7e9d2... 2023-06-23T01:23:38.031 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T01:23:38.072 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T01:23:38.116 INFO:teuthology.orchestra.run.smithi193.stdout:[global] 2023-06-23T01:23:38.116 INFO:teuthology.orchestra.run.smithi193.stdout: chdir = "" 2023-06-23T01:23:38.117 INFO:teuthology.orchestra.run.smithi193.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T01:23:38.117 INFO:teuthology.orchestra.run.smithi193.stdout: auth supported = cephx 2023-06-23T01:23:38.117 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.117 INFO:teuthology.orchestra.run.smithi193.stdout: filestore xattr use omap = true 2023-06-23T01:23:38.118 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.118 INFO:teuthology.orchestra.run.smithi193.stdout: mon clock drift allowed = 1.000 2023-06-23T01:23:38.118 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.118 INFO:teuthology.orchestra.run.smithi193.stdout: osd crush chooseleaf type = 0 2023-06-23T01:23:38.118 INFO:teuthology.orchestra.run.smithi193.stdout: auth debug = true 2023-06-23T01:23:38.118 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.119 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on old message = true 2023-06-23T01:23:38.119 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on bug = true 2023-06-23T01:23:38.119 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.119 INFO:teuthology.orchestra.run.smithi193.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T01:23:38.119 INFO:teuthology.orchestra.run.smithi193.stdout: mon pg warn max object skew = 0 2023-06-23T01:23:38.119 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.119 INFO:teuthology.orchestra.run.smithi193.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T01:23:38.119 INFO:teuthology.orchestra.run.smithi193.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T01:23:38.120 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.120 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default size = 2 2023-06-23T01:23:38.120 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.120 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow primary affinity = true 2023-06-23T01:23:38.120 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd allow pg remap = true 2023-06-23T01:23:38.120 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on legacy crush tunables = false 2023-06-23T01:23:38.120 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on crush straw calc version zero = false 2023-06-23T01:23:38.121 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on no sortbitwise = false 2023-06-23T01:23:38.121 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on osd down out interval zero = false 2023-06-23T01:23:38.121 INFO:teuthology.orchestra.run.smithi193.stdout: mon warn on too few osds = false 2023-06-23T01:23:38.121 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T01:23:38.121 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T01:23:38.121 INFO:teuthology.orchestra.run.smithi193.stdout: mon_allow_pool_size_one = true 2023-06-23T01:23:38.122 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.122 INFO:teuthology.orchestra.run.smithi193.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T01:23:38.122 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.122 INFO:teuthology.orchestra.run.smithi193.stdout: osd default data pool replay window = 5 2023-06-23T01:23:38.122 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.123 INFO:teuthology.orchestra.run.smithi193.stdout: mon allow pool delete = true 2023-06-23T01:23:38.123 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.124 INFO:teuthology.orchestra.run.smithi193.stdout: mon cluster log file level = debug 2023-06-23T01:23:38.124 INFO:teuthology.orchestra.run.smithi193.stdout: debug asserts on shutdown = true 2023-06-23T01:23:38.125 INFO:teuthology.orchestra.run.smithi193.stdout: mon health detail to clog = false 2023-06-23T01:23:38.126 INFO:teuthology.orchestra.run.smithi193.stdout: mon host = "172.21.15.67,172.21.15.193,[v2:172.21.15.193:3301,v1:172.21.15.193:6790]" 2023-06-23T01:23:38.126 INFO:teuthology.orchestra.run.smithi193.stdout: ms die on skipped message = False 2023-06-23T01:23:38.126 INFO:teuthology.orchestra.run.smithi193.stdout: fsid = 0068cd93-d7dd-4502-a85d-c8368ea7e9d2 2023-06-23T01:23:38.126 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.127 INFO:teuthology.orchestra.run.smithi193.stdout:[osd] 2023-06-23T01:23:38.127 INFO:teuthology.orchestra.run.smithi193.stdout: osd journal size = 100 2023-06-23T01:23:38.127 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.127 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub load threshold = 5.0 2023-06-23T01:23:38.127 INFO:teuthology.orchestra.run.smithi193.stdout: osd scrub max interval = 600 2023-06-23T01:23:38.127 INFO:teuthology.orchestra.run.smithi193.stdout: osd mclock profile = high_recovery_ops 2023-06-23T01:23:38.127 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.128 INFO:teuthology.orchestra.run.smithi193.stdout: osd recover clone overlap = true 2023-06-23T01:23:38.128 INFO:teuthology.orchestra.run.smithi193.stdout: osd recovery max chunk = 1048576 2023-06-23T01:23:38.128 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.128 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug shutdown = true 2023-06-23T01:23:38.128 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug op order = true 2023-06-23T01:23:38.128 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug verify stray on activate = true 2023-06-23T01:23:38.128 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.129 INFO:teuthology.orchestra.run.smithi193.stdout: osd open classes on start = true 2023-06-23T01:23:38.129 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug pg log writeout = true 2023-06-23T01:23:38.129 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.129 INFO:teuthology.orchestra.run.smithi193.stdout: osd deep scrub update digest min age = 30 2023-06-23T01:23:38.129 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.129 INFO:teuthology.orchestra.run.smithi193.stdout: osd map max advance = 10 2023-06-23T01:23:38.129 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.130 INFO:teuthology.orchestra.run.smithi193.stdout: journal zero on create = true 2023-06-23T01:23:38.130 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.130 INFO:teuthology.orchestra.run.smithi193.stdout: filestore ondisk finisher threads = 3 2023-06-23T01:23:38.130 INFO:teuthology.orchestra.run.smithi193.stdout: filestore apply finisher threads = 3 2023-06-23T01:23:38.130 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.130 INFO:teuthology.orchestra.run.smithi193.stdout: bdev debug aio = true 2023-06-23T01:23:38.130 INFO:teuthology.orchestra.run.smithi193.stdout: osd debug misdirected ops = true 2023-06-23T01:23:38.131 INFO:teuthology.orchestra.run.smithi193.stdout: bdev async discard = True 2023-06-23T01:23:38.131 INFO:teuthology.orchestra.run.smithi193.stdout: bdev enable discard = True 2023-06-23T01:23:38.131 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore allocator = bitmap 2023-06-23T01:23:38.131 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore block size = 96636764160 2023-06-23T01:23:38.131 INFO:teuthology.orchestra.run.smithi193.stdout: bluestore fsck on mount = True 2023-06-23T01:23:38.131 INFO:teuthology.orchestra.run.smithi193.stdout: debug bluefs = 1/20 2023-06-23T01:23:38.131 INFO:teuthology.orchestra.run.smithi193.stdout: debug bluestore = 1/20 2023-06-23T01:23:38.132 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-06-23T01:23:38.132 INFO:teuthology.orchestra.run.smithi193.stdout: debug osd = 20 2023-06-23T01:23:38.132 INFO:teuthology.orchestra.run.smithi193.stdout: debug rocksdb = 4/10 2023-06-23T01:23:38.132 INFO:teuthology.orchestra.run.smithi193.stdout: memstore device bytes = 200000000 2023-06-23T01:23:38.132 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T01:23:38.132 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd full ratio = 0.9 2023-06-23T01:23:38.132 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd nearfull ratio = 0.8 2023-06-23T01:23:38.133 INFO:teuthology.orchestra.run.smithi193.stdout: osd failsafe full ratio = 1.0 2023-06-23T01:23:38.133 INFO:teuthology.orchestra.run.smithi193.stdout: osd mon report interval = 5 2023-06-23T01:23:38.133 INFO:teuthology.orchestra.run.smithi193.stdout: osd objectstore = memstore 2023-06-23T01:23:38.133 INFO:teuthology.orchestra.run.smithi193.stdout: osd op complaint time = 180 2023-06-23T01:23:38.133 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.133 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr] 2023-06-23T01:23:38.133 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-06-23T01:23:38.134 INFO:teuthology.orchestra.run.smithi193.stdout: debug mgr = 20 2023-06-23T01:23:38.134 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2023-06-23T01:23:38.134 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2023-06-23T01:23:38.134 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:23:38.134 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:23:38.134 INFO:teuthology.orchestra.run.smithi193.stdout: mgr/telemetry/nag = false 2023-06-23T01:23:38.134 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.135 INFO:teuthology.orchestra.run.smithi193.stdout:[mon] 2023-06-23T01:23:38.135 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-06-23T01:23:38.135 INFO:teuthology.orchestra.run.smithi193.stdout: debug mon = 20 2023-06-23T01:23:38.135 INFO:teuthology.orchestra.run.smithi193.stdout: debug paxos = 20 2023-06-23T01:23:38.135 INFO:teuthology.orchestra.run.smithi193.stdout: debug auth = 20 2023-06-23T01:23:38.135 INFO:teuthology.orchestra.run.smithi193.stdout: mon data avail warn = 5 2023-06-23T01:23:38.135 INFO:teuthology.orchestra.run.smithi193.stdout: mon mgr mkfs grace = 240 2023-06-23T01:23:38.136 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:23:38.136 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd reporter subtree level = osd 2023-06-23T01:23:38.136 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd prime pg temp = true 2023-06-23T01:23:38.136 INFO:teuthology.orchestra.run.smithi193.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:23:38.136 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.136 INFO:teuthology.orchestra.run.smithi193.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T01:23:38.136 INFO:teuthology.orchestra.run.smithi193.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T01:23:38.137 INFO:teuthology.orchestra.run.smithi193.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T01:23:38.137 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.137 INFO:teuthology.orchestra.run.smithi193.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T01:23:38.137 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T01:23:38.137 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T01:23:38.137 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.137 INFO:teuthology.orchestra.run.smithi193.stdout: # 1m isn't quite enough 2023-06-23T01:23:38.138 INFO:teuthology.orchestra.run.smithi193.stdout: mon_down_mkfs_grace = 2m 2023-06-23T01:23:38.138 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.138 INFO:teuthology.orchestra.run.smithi193.stdout: mon_warn_on_filestore_osds = false 2023-06-23T01:23:38.138 INFO:teuthology.orchestra.run.smithi193.stdout: mon op complaint time = 120 2023-06-23T01:23:38.138 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd backfillfull ratio = 0.6 2023-06-23T01:23:38.138 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd full ratio = 0.7 2023-06-23T01:23:38.138 INFO:teuthology.orchestra.run.smithi193.stdout: mon osd nearfull ratio = 0.6 2023-06-23T01:23:38.139 INFO:teuthology.orchestra.run.smithi193.stdout: 2023-06-23T01:23:38.139 INFO:teuthology.orchestra.run.smithi193.stdout:[client] 2023-06-23T01:23:38.139 INFO:teuthology.orchestra.run.smithi193.stdout: rgw cache enabled = true 2023-06-23T01:23:38.139 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable ops log = true 2023-06-23T01:23:38.139 INFO:teuthology.orchestra.run.smithi193.stdout: rgw enable usage log = true 2023-06-23T01:23:38.139 INFO:teuthology.orchestra.run.smithi193.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T01:23:38.139 INFO:teuthology.orchestra.run.smithi193.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T01:23:38.140 INFO:teuthology.orchestra.run.smithi193.stdout: client mount timeout = 600 2023-06-23T01:23:38.140 INFO:teuthology.orchestra.run.smithi193.stdout: debug client = 20 2023-06-23T01:23:38.140 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-06-23T01:23:38.140 INFO:teuthology.orchestra.run.smithi193.stdout: pid file = "" 2023-06-23T01:23:38.140 INFO:teuthology.orchestra.run.smithi193.stdout: rados mon op timeout = 900 2023-06-23T01:23:38.140 INFO:teuthology.orchestra.run.smithi193.stdout: rados osd op timeout = 900 2023-06-23T01:23:38.140 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.a] 2023-06-23T01:23:38.141 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.b] 2023-06-23T01:23:38.141 INFO:teuthology.orchestra.run.smithi193.stdout:[mon.c] 2023-06-23T01:23:38.141 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.a] 2023-06-23T01:23:38.141 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.c] 2023-06-23T01:23:38.141 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.b] 2023-06-23T01:23:38.141 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.d] 2023-06-23T01:23:38.141 INFO:teuthology.orchestra.run.smithi193.stdout:[client.0] 2023-06-23T01:23:38.141 INFO:teuthology.orchestra.run.smithi193.stdout: debug client = 20 2023-06-23T01:23:38.142 INFO:teuthology.orchestra.run.smithi193.stdout: debug objectcacher = 20 2023-06-23T01:23:38.142 INFO:teuthology.orchestra.run.smithi193.stdout: debug objecter = 20 2023-06-23T01:23:38.142 INFO:teuthology.orchestra.run.smithi193.stdout:[client.1] 2023-06-23T01:23:38.142 INFO:teuthology.orchestra.run.smithi193.stdout: debug client = 20 2023-06-23T01:23:38.142 INFO:teuthology.orchestra.run.smithi193.stdout: debug objectcacher = 20 2023-06-23T01:23:38.142 INFO:teuthology.orchestra.run.smithi193.stdout: debug objecter = 20 2023-06-23T01:23:38.142 INFO:teuthology.orchestra.run.smithi193.stdout:[mds] 2023-06-23T01:23:38.143 INFO:teuthology.orchestra.run.smithi193.stdout: debug mds = 20 2023-06-23T01:23:38.143 INFO:teuthology.orchestra.run.smithi193.stdout: debug ms = 1 2023-06-23T01:23:38.143 INFO:teuthology.orchestra.run.smithi193.stdout: mds debug frag = True 2023-06-23T01:23:38.143 INFO:teuthology.orchestra.run.smithi193.stdout: mds debug scatterstat = True 2023-06-23T01:23:38.143 INFO:teuthology.orchestra.run.smithi193.stdout: mds op complaint time = 180 2023-06-23T01:23:38.143 INFO:teuthology.orchestra.run.smithi193.stdout: mds verify scatter = True 2023-06-23T01:23:38.143 INFO:teuthology.orchestra.run.smithi193.stdout: osd op complaint time = 180 2023-06-23T01:23:38.144 INFO:teuthology.orchestra.run.smithi193.stdout: rados mon op timeout = 900 2023-06-23T01:23:38.144 INFO:teuthology.orchestra.run.smithi193.stdout: rados osd op timeout = 900 2023-06-23T01:23:38.147 INFO:teuthology.orchestra.run.smithi067.stdout:[global] 2023-06-23T01:23:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: chdir = "" 2023-06-23T01:23:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T01:23:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: auth supported = cephx 2023-06-23T01:23:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: filestore xattr use omap = true 2023-06-23T01:23:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.148 INFO:teuthology.orchestra.run.smithi067.stdout: mon clock drift allowed = 1.000 2023-06-23T01:23:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: osd crush chooseleaf type = 0 2023-06-23T01:23:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: auth debug = true 2023-06-23T01:23:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on old message = true 2023-06-23T01:23:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on bug = true 2023-06-23T01:23:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.149 INFO:teuthology.orchestra.run.smithi067.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T01:23:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: mon pg warn max object skew = 0 2023-06-23T01:23:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T01:23:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T01:23:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default size = 2 2023-06-23T01:23:38.150 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow primary affinity = true 2023-06-23T01:23:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd allow pg remap = true 2023-06-23T01:23:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on legacy crush tunables = false 2023-06-23T01:23:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on crush straw calc version zero = false 2023-06-23T01:23:38.151 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on no sortbitwise = false 2023-06-23T01:23:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on osd down out interval zero = false 2023-06-23T01:23:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: mon warn on too few osds = false 2023-06-23T01:23:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T01:23:38.152 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T01:23:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: mon_allow_pool_size_one = true 2023-06-23T01:23:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.153 INFO:teuthology.orchestra.run.smithi067.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T01:23:38.156 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.156 INFO:teuthology.orchestra.run.smithi067.stdout: osd default data pool replay window = 5 2023-06-23T01:23:38.157 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.157 INFO:teuthology.orchestra.run.smithi067.stdout: mon allow pool delete = true 2023-06-23T01:23:38.157 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.157 INFO:teuthology.orchestra.run.smithi067.stdout: mon cluster log file level = debug 2023-06-23T01:23:38.157 INFO:teuthology.orchestra.run.smithi067.stdout: debug asserts on shutdown = true 2023-06-23T01:23:38.157 INFO:teuthology.orchestra.run.smithi067.stdout: mon health detail to clog = false 2023-06-23T01:23:38.157 INFO:teuthology.orchestra.run.smithi067.stdout: mon host = "172.21.15.67,172.21.15.193,[v2:172.21.15.193:3301,v1:172.21.15.193:6790]" 2023-06-23T01:23:38.157 INFO:teuthology.orchestra.run.smithi067.stdout: ms die on skipped message = False 2023-06-23T01:23:38.158 INFO:teuthology.orchestra.run.smithi067.stdout: fsid = 0068cd93-d7dd-4502-a85d-c8368ea7e9d2 2023-06-23T01:23:38.158 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.158 INFO:teuthology.orchestra.run.smithi067.stdout:[osd] 2023-06-23T01:23:38.158 INFO:teuthology.orchestra.run.smithi067.stdout: osd journal size = 100 2023-06-23T01:23:38.158 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.158 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub load threshold = 5.0 2023-06-23T01:23:38.158 INFO:teuthology.orchestra.run.smithi067.stdout: osd scrub max interval = 600 2023-06-23T01:23:38.159 INFO:teuthology.orchestra.run.smithi067.stdout: osd mclock profile = high_recovery_ops 2023-06-23T01:23:38.159 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.159 INFO:teuthology.orchestra.run.smithi067.stdout: osd recover clone overlap = true 2023-06-23T01:23:38.159 INFO:teuthology.orchestra.run.smithi067.stdout: osd recovery max chunk = 1048576 2023-06-23T01:23:38.159 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.159 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug shutdown = true 2023-06-23T01:23:38.159 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug op order = true 2023-06-23T01:23:38.160 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug verify stray on activate = true 2023-06-23T01:23:38.160 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.160 INFO:teuthology.orchestra.run.smithi067.stdout: osd open classes on start = true 2023-06-23T01:23:38.160 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug pg log writeout = true 2023-06-23T01:23:38.160 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.160 INFO:teuthology.orchestra.run.smithi067.stdout: osd deep scrub update digest min age = 30 2023-06-23T01:23:38.160 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.161 INFO:teuthology.orchestra.run.smithi067.stdout: osd map max advance = 10 2023-06-23T01:23:38.161 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.161 INFO:teuthology.orchestra.run.smithi067.stdout: journal zero on create = true 2023-06-23T01:23:38.161 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.161 INFO:teuthology.orchestra.run.smithi067.stdout: filestore ondisk finisher threads = 3 2023-06-23T01:23:38.161 INFO:teuthology.orchestra.run.smithi067.stdout: filestore apply finisher threads = 3 2023-06-23T01:23:38.161 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.161 INFO:teuthology.orchestra.run.smithi067.stdout: bdev debug aio = true 2023-06-23T01:23:38.162 INFO:teuthology.orchestra.run.smithi067.stdout: osd debug misdirected ops = true 2023-06-23T01:23:38.162 INFO:teuthology.orchestra.run.smithi067.stdout: bdev async discard = True 2023-06-23T01:23:38.162 INFO:teuthology.orchestra.run.smithi067.stdout: bdev enable discard = True 2023-06-23T01:23:38.162 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore allocator = bitmap 2023-06-23T01:23:38.162 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore block size = 96636764160 2023-06-23T01:23:38.162 INFO:teuthology.orchestra.run.smithi067.stdout: bluestore fsck on mount = True 2023-06-23T01:23:38.162 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluefs = 1/20 2023-06-23T01:23:38.162 INFO:teuthology.orchestra.run.smithi067.stdout: debug bluestore = 1/20 2023-06-23T01:23:38.163 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-23T01:23:38.163 INFO:teuthology.orchestra.run.smithi067.stdout: debug osd = 20 2023-06-23T01:23:38.163 INFO:teuthology.orchestra.run.smithi067.stdout: debug rocksdb = 4/10 2023-06-23T01:23:38.163 INFO:teuthology.orchestra.run.smithi067.stdout: memstore device bytes = 200000000 2023-06-23T01:23:38.163 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T01:23:38.163 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.9 2023-06-23T01:23:38.163 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.8 2023-06-23T01:23:38.163 INFO:teuthology.orchestra.run.smithi067.stdout: osd failsafe full ratio = 1.0 2023-06-23T01:23:38.164 INFO:teuthology.orchestra.run.smithi067.stdout: osd mon report interval = 5 2023-06-23T01:23:38.164 INFO:teuthology.orchestra.run.smithi067.stdout: osd objectstore = memstore 2023-06-23T01:23:38.164 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2023-06-23T01:23:38.164 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.164 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr] 2023-06-23T01:23:38.164 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-23T01:23:38.164 INFO:teuthology.orchestra.run.smithi067.stdout: debug mgr = 20 2023-06-23T01:23:38.165 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2023-06-23T01:23:38.165 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2023-06-23T01:23:38.165 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:23:38.165 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:23:38.165 INFO:teuthology.orchestra.run.smithi067.stdout: mgr/telemetry/nag = false 2023-06-23T01:23:38.165 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.165 INFO:teuthology.orchestra.run.smithi067.stdout:[mon] 2023-06-23T01:23:38.165 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-23T01:23:38.166 INFO:teuthology.orchestra.run.smithi067.stdout: debug mon = 20 2023-06-23T01:23:38.166 INFO:teuthology.orchestra.run.smithi067.stdout: debug paxos = 20 2023-06-23T01:23:38.166 INFO:teuthology.orchestra.run.smithi067.stdout: debug auth = 20 2023-06-23T01:23:38.166 INFO:teuthology.orchestra.run.smithi067.stdout: mon data avail warn = 5 2023-06-23T01:23:38.166 INFO:teuthology.orchestra.run.smithi067.stdout: mon mgr mkfs grace = 240 2023-06-23T01:23:38.166 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:23:38.166 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd reporter subtree level = osd 2023-06-23T01:23:38.166 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd prime pg temp = true 2023-06-23T01:23:38.167 INFO:teuthology.orchestra.run.smithi067.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:23:38.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.167 INFO:teuthology.orchestra.run.smithi067.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T01:23:38.167 INFO:teuthology.orchestra.run.smithi067.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T01:23:38.167 INFO:teuthology.orchestra.run.smithi067.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T01:23:38.167 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.167 INFO:teuthology.orchestra.run.smithi067.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T01:23:38.168 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T01:23:38.168 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T01:23:38.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.168 INFO:teuthology.orchestra.run.smithi067.stdout: # 1m isn't quite enough 2023-06-23T01:23:38.168 INFO:teuthology.orchestra.run.smithi067.stdout: mon_down_mkfs_grace = 2m 2023-06-23T01:23:38.168 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.168 INFO:teuthology.orchestra.run.smithi067.stdout: mon_warn_on_filestore_osds = false 2023-06-23T01:23:38.168 INFO:teuthology.orchestra.run.smithi067.stdout: mon op complaint time = 120 2023-06-23T01:23:38.169 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd backfillfull ratio = 0.6 2023-06-23T01:23:38.169 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd full ratio = 0.7 2023-06-23T01:23:38.169 INFO:teuthology.orchestra.run.smithi067.stdout: mon osd nearfull ratio = 0.6 2023-06-23T01:23:38.169 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:38.169 INFO:teuthology.orchestra.run.smithi067.stdout:[client] 2023-06-23T01:23:38.169 INFO:teuthology.orchestra.run.smithi067.stdout: rgw cache enabled = true 2023-06-23T01:23:38.169 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable ops log = true 2023-06-23T01:23:38.169 INFO:teuthology.orchestra.run.smithi067.stdout: rgw enable usage log = true 2023-06-23T01:23:38.170 INFO:teuthology.orchestra.run.smithi067.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T01:23:38.170 INFO:teuthology.orchestra.run.smithi067.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T01:23:38.170 INFO:teuthology.orchestra.run.smithi067.stdout: client mount timeout = 600 2023-06-23T01:23:38.170 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2023-06-23T01:23:38.170 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-23T01:23:38.170 INFO:teuthology.orchestra.run.smithi067.stdout: pid file = "" 2023-06-23T01:23:38.170 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2023-06-23T01:23:38.170 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2023-06-23T01:23:38.171 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.a] 2023-06-23T01:23:38.171 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.b] 2023-06-23T01:23:38.171 INFO:teuthology.orchestra.run.smithi067.stdout:[mon.c] 2023-06-23T01:23:38.171 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2023-06-23T01:23:38.171 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2023-06-23T01:23:38.171 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2023-06-23T01:23:38.171 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2023-06-23T01:23:38.172 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2023-06-23T01:23:38.172 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2023-06-23T01:23:38.172 INFO:teuthology.orchestra.run.smithi067.stdout: debug objectcacher = 20 2023-06-23T01:23:38.172 INFO:teuthology.orchestra.run.smithi067.stdout: debug objecter = 20 2023-06-23T01:23:38.172 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2023-06-23T01:23:38.172 INFO:teuthology.orchestra.run.smithi067.stdout: debug client = 20 2023-06-23T01:23:38.173 INFO:teuthology.orchestra.run.smithi067.stdout: debug objectcacher = 20 2023-06-23T01:23:38.173 INFO:teuthology.orchestra.run.smithi067.stdout: debug objecter = 20 2023-06-23T01:23:38.173 INFO:teuthology.orchestra.run.smithi067.stdout:[mds] 2023-06-23T01:23:38.173 INFO:teuthology.orchestra.run.smithi067.stdout: debug mds = 20 2023-06-23T01:23:38.173 INFO:teuthology.orchestra.run.smithi067.stdout: debug ms = 1 2023-06-23T01:23:38.173 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug frag = True 2023-06-23T01:23:38.173 INFO:teuthology.orchestra.run.smithi067.stdout: mds debug scatterstat = True 2023-06-23T01:23:38.174 INFO:teuthology.orchestra.run.smithi067.stdout: mds op complaint time = 180 2023-06-23T01:23:38.174 INFO:teuthology.orchestra.run.smithi067.stdout: mds verify scatter = True 2023-06-23T01:23:38.174 INFO:teuthology.orchestra.run.smithi067.stdout: osd op complaint time = 180 2023-06-23T01:23:38.174 INFO:teuthology.orchestra.run.smithi067.stdout: rados mon op timeout = 900 2023-06-23T01:23:38.174 INFO:teuthology.orchestra.run.smithi067.stdout: rados osd op timeout = 900 2023-06-23T01:23:38.175 INFO:tasks.ceph:Creating admin key on mon.a... 2023-06-23T01:23:38.175 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-06-23T01:23:38.249 INFO:tasks.ceph:Copying monmap to all nodes... 2023-06-23T01:23:38.249 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:38.250 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-06-23T01:23:38.265 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:38.265 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-06-23T01:23:38.322 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:23:38.323 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:38.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T01:23:38.324 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:23:38.397 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:38.398 DEBUG:teuthology.orchestra.run.smithi067:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:38.453 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:23:38.453 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:38.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T01:23:38.453 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:23:38.487 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:38.488 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:38.541 INFO:tasks.ceph:Setting up mon nodes... 2023-06-23T01:23:38.542 INFO:tasks.ceph:Setting up mgr nodes... 2023-06-23T01:23:38.542 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T01:23:38.593 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T01:23:38.595 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T01:23:38.663 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T01:23:38.666 INFO:tasks.ceph:Setting up mds nodes... 2023-06-23T01:23:38.666 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T01:23:38.717 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T01:23:38.719 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-06-23T01:23:38.744 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T01:23:38.836 INFO:teuthology.orchestra.run.smithi067.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T01:23:38.838 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-06-23T01:23:38.864 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T01:23:38.913 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T01:23:38.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-06-23T01:23:38.940 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T01:23:39.027 INFO:teuthology.orchestra.run.smithi193.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T01:23:39.030 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-06-23T01:23:39.060 INFO:tasks.ceph_client:Setting up client nodes... 2023-06-23T01:23:39.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-06-23T01:23:39.100 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-06-23T01:23:39.113 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-06-23T01:23:39.193 INFO:teuthology.orchestra.run.smithi067.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-06-23T01:23:39.207 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-06-23T01:23:39.248 INFO:teuthology.orchestra.run.smithi193.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-06-23T01:23:39.257 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-06-23T01:23:39.335 INFO:teuthology.orchestra.run.smithi193.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-06-23T01:23:39.347 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-06-23T01:23:39.347 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi067.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi193.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T01:23:39.348 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-06-23T01:23:39.374 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:39.374 INFO:tasks.ceph:role: osd.0 2023-06-23T01:23:39.374 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:23:39.375 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T01:23:39.448 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:23:39.448 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:23:39.449 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:23:39.449 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 2023-06-23T01:23:39.449 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:23:39.449 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:23:39.449 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:23:39.449 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:23:39.449 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:23:39.450 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:23:39.869 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-06-23T01:23:39.870 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-06-23T01:23:39.871 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-06-23T01:23:40.029 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-06-23T01:23:40.061 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-06-23T01:23:40.128 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:40.128 INFO:tasks.ceph:role: osd.1 2023-06-23T01:23:40.129 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:23:40.129 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T01:23:40.201 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:23:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:23:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:23:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 2023-06-23T01:23:40.202 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:23:40.202 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:23:40.203 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:23:40.203 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:23:40.203 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:23:40.203 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:23:40.636 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-06-23T01:23:40.638 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-06-23T01:23:40.638 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-06-23T01:23:40.740 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-06-23T01:23:40.768 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-06-23T01:23:40.836 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:40.836 INFO:tasks.ceph:role: osd.2 2023-06-23T01:23:40.837 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:23:40.838 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T01:23:40.908 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:23:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:23:40.909 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:23:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 2023-06-23T01:23:40.910 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:23:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:23:40.910 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:23:40.910 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:23:40.910 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:23:40.911 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:23:41.352 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-06-23T01:23:41.354 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-06-23T01:23:41.354 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-06-23T01:23:41.506 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-06-23T01:23:41.538 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-06-23T01:23:41.603 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:41.603 INFO:tasks.ceph:role: osd.3 2023-06-23T01:23:41.603 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:23:41.604 DEBUG:teuthology.orchestra.run.smithi067:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T01:23:41.675 INFO:teuthology.orchestra.run.smithi067.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:23:41.676 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:23:41.676 INFO:teuthology.orchestra.run.smithi067.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:23:41.676 INFO:teuthology.orchestra.run.smithi067.stdout: = reflink=1 2023-06-23T01:23:41.676 INFO:teuthology.orchestra.run.smithi067.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:23:41.677 INFO:teuthology.orchestra.run.smithi067.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:23:41.677 INFO:teuthology.orchestra.run.smithi067.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:23:41.677 INFO:teuthology.orchestra.run.smithi067.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:23:41.678 INFO:teuthology.orchestra.run.smithi067.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:23:41.678 INFO:teuthology.orchestra.run.smithi067.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:23:42.118 INFO:teuthology.orchestra.run.smithi067.stdout:Discarding blocks...Done. 2023-06-23T01:23:42.121 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi067.front.sepia.ceph.com -o noatime 2023-06-23T01:23:42.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-06-23T01:23:42.272 DEBUG:teuthology.orchestra.run.smithi067:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-06-23T01:23:42.304 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:42.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.429+0000 7fe2bbd94540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-06-23T01:23:42.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.429+0000 7fe2bbd94540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-06-23T01:23:42.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.429+0000 7fe2bbd94540 -1 memstore(/var/lib/ceph/osd/ceph-0) /var/lib/ceph/osd/ceph-0 2023-06-23T01:23:42.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-06-23T01:23:42.469 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:42.550 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.547+0000 7fa706a7b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-06-23T01:23:42.551 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.548+0000 7fa706a7b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-06-23T01:23:42.552 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.549+0000 7fa706a7b540 -1 memstore(/var/lib/ceph/osd/ceph-1) /var/lib/ceph/osd/ceph-1 2023-06-23T01:23:42.558 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-06-23T01:23:42.624 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:42.705 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.702+0000 7fe05c641540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-06-23T01:23:42.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.702+0000 7fe05c641540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-06-23T01:23:42.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.703+0000 7fe05c641540 -1 memstore(/var/lib/ceph/osd/ceph-2) /var/lib/ceph/osd/ceph-2 2023-06-23T01:23:42.717 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-06-23T01:23:42.782 DEBUG:teuthology.orchestra.run.smithi067:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.913+0000 7fe69ebba540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-06-23T01:23:42.916 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.913+0000 7fe69ebba540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-06-23T01:23:42.917 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:42.913+0000 7fe69ebba540 -1 memstore(/var/lib/ceph/osd/ceph-3) /var/lib/ceph/osd/ceph-3 2023-06-23T01:23:42.922 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-06-23T01:23:42.947 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-06-23T01:23:42.977 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:42.977 INFO:tasks.ceph:role: osd.4 2023-06-23T01:23:42.977 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:23:42.977 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T01:23:43.051 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:23:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:23:43.051 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:23:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 2023-06-23T01:23:43.052 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:23:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:23:43.052 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:23:43.052 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:23:43.052 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:23:43.052 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:23:43.484 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2023-06-23T01:23:43.486 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2023-06-23T01:23:43.487 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-06-23T01:23:43.659 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-06-23T01:23:43.693 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-06-23T01:23:43.761 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:43.761 INFO:tasks.ceph:role: osd.5 2023-06-23T01:23:43.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:23:43.761 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T01:23:43.834 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:23:43.834 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:23:43.834 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:23:43.834 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 2023-06-23T01:23:43.835 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:23:43.835 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:23:43.835 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:23:43.835 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:23:43.835 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:23:43.836 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:23:44.246 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2023-06-23T01:23:44.248 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2023-06-23T01:23:44.249 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-06-23T01:23:44.363 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-06-23T01:23:44.391 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-06-23T01:23:44.456 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:44.456 INFO:tasks.ceph:role: osd.6 2023-06-23T01:23:44.457 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:23:44.457 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T01:23:44.528 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:23:44.528 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:23:44.529 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:23:44.529 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 2023-06-23T01:23:44.529 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:23:44.529 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:23:44.529 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:23:44.529 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:23:44.530 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:23:44.530 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:23:44.980 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2023-06-23T01:23:44.982 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2023-06-23T01:23:44.982 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-06-23T01:23:45.139 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-06-23T01:23:45.169 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-06-23T01:23:45.233 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:23:45.233 INFO:tasks.ceph:role: osd.7 2023-06-23T01:23:45.234 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:23:45.234 DEBUG:teuthology.orchestra.run.smithi193:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T01:23:45.305 INFO:teuthology.orchestra.run.smithi193.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:23:45.305 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:23:45.306 INFO:teuthology.orchestra.run.smithi193.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:23:45.306 INFO:teuthology.orchestra.run.smithi193.stdout: = reflink=1 2023-06-23T01:23:45.306 INFO:teuthology.orchestra.run.smithi193.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:23:45.306 INFO:teuthology.orchestra.run.smithi193.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:23:45.307 INFO:teuthology.orchestra.run.smithi193.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:23:45.307 INFO:teuthology.orchestra.run.smithi193.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:23:45.307 INFO:teuthology.orchestra.run.smithi193.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:23:45.307 INFO:teuthology.orchestra.run.smithi193.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:23:45.745 INFO:teuthology.orchestra.run.smithi193.stdout:Discarding blocks...Done. 2023-06-23T01:23:45.747 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi193.front.sepia.ceph.com -o noatime 2023-06-23T01:23:45.747 DEBUG:teuthology.orchestra.run.smithi193:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-06-23T01:23:45.900 DEBUG:teuthology.orchestra.run.smithi193:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-06-23T01:23:45.932 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:46.020 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.017+0000 7f53332cc540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-06-23T01:23:46.021 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.019+0000 7f53332cc540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-06-23T01:23:46.021 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.019+0000 7f53332cc540 -1 memstore(/var/lib/ceph/osd/ceph-4) /var/lib/ceph/osd/ceph-4 2023-06-23T01:23:46.031 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-06-23T01:23:46.055 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:46.135 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.132+0000 7fc11e1d9540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-06-23T01:23:46.135 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.133+0000 7fc11e1d9540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-06-23T01:23:46.136 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.133+0000 7fc11e1d9540 -1 memstore(/var/lib/ceph/osd/ceph-5) /var/lib/ceph/osd/ceph-5 2023-06-23T01:23:46.141 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-06-23T01:23:46.205 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:46.285 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.283+0000 7f1639777540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-06-23T01:23:46.285 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.283+0000 7f1639777540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-06-23T01:23:46.286 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.283+0000 7f1639777540 -1 memstore(/var/lib/ceph/osd/ceph-6) /var/lib/ceph/osd/ceph-6 2023-06-23T01:23:46.294 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-06-23T01:23:46.359 DEBUG:teuthology.orchestra.run.smithi193:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:46.438 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.435+0000 7f40404c1540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-06-23T01:23:46.439 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.435+0000 7f40404c1540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-06-23T01:23:46.439 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:46.436+0000 7f40404c1540 -1 memstore(/var/lib/ceph/osd/ceph-7) /var/lib/ceph/osd/ceph-7 2023-06-23T01:23:46.444 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-06-23T01:23:46.511 INFO:tasks.ceph:Reading keys from all nodes... 2023-06-23T01:23:46.512 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:46.512 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-06-23T01:23:46.538 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:46.539 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-06-23T01:23:46.603 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:46.603 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-06-23T01:23:46.668 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:46.668 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-06-23T01:23:46.733 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:46.733 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-06-23T01:23:46.798 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:46.799 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-06-23T01:23:46.865 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:46.865 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-06-23T01:23:46.930 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:46.930 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-06-23T01:23:46.954 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:46.955 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-06-23T01:23:47.018 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:47.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-06-23T01:23:47.082 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:47.082 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-06-23T01:23:47.145 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:47.146 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-06-23T01:23:47.209 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:47.210 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-06-23T01:23:47.273 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:47.273 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-06-23T01:23:47.336 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:47.337 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-06-23T01:23:47.351 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:47.352 DEBUG:teuthology.orchestra.run.smithi067:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-06-23T01:23:47.406 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:47.406 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-06-23T01:23:47.420 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:47.421 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-06-23T01:23:47.475 INFO:tasks.ceph:Adding keys to all mons... 2023-06-23T01:23:47.475 DEBUG:teuthology.orchestra.run.smithi067:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T01:23:47.477 DEBUG:teuthology.orchestra.run.smithi193:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T01:23:47.518 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.y] 2023-06-23T01:23:47.518 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAa9JRkxjpFIxAAJfAa076whBzTR7UBFd/DEA== 2023-06-23T01:23:47.519 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.a] 2023-06-23T01:23:47.519 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAa9JRkUmSrKhAAj+j81xXc4pyIEb/FSzbxZg== 2023-06-23T01:23:47.519 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.c] 2023-06-23T01:23:47.519 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAa9JRkdUTBMRAAQHT0bf2rORbANYJYWUHLQQ== 2023-06-23T01:23:47.519 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.0] 2023-06-23T01:23:47.519 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAe9JRkKZOtGRAA571AepEUjHUQFgGjs6cxEQ== 2023-06-23T01:23:47.519 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.1] 2023-06-23T01:23:47.520 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAe9JRkouG9IBAAzSh7OwpoFejMHaduJ7S/Dw== 2023-06-23T01:23:47.520 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.2] 2023-06-23T01:23:47.520 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAe9JRkxqv3KRAAgZ7hzl9+KmMidcD5OAzHPw== 2023-06-23T01:23:47.520 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.3] 2023-06-23T01:23:47.520 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAe9JRkNYyENhAAcRB/FK4kxcIeU7Jb4nvE3g== 2023-06-23T01:23:47.520 INFO:teuthology.orchestra.run.smithi067.stdout:[mgr.x] 2023-06-23T01:23:47.521 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAa9JRkkvBwJxAA6xoW9R0SsDXcrMlRilTiLw== 2023-06-23T01:23:47.521 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.b] 2023-06-23T01:23:47.521 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAa9JRkjnZcNhAAEs+mQw+Igz2nS4r3YvDJ4A== 2023-06-23T01:23:47.521 INFO:teuthology.orchestra.run.smithi067.stdout:[mds.d] 2023-06-23T01:23:47.521 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAb9JRkffCPARAAQtAdBnWbfxHiN/G+9OQnyA== 2023-06-23T01:23:47.522 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.4] 2023-06-23T01:23:47.522 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAi9JRkxjMdARAA6YDqLhW7Q3AwdiDfnxBTrQ== 2023-06-23T01:23:47.522 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.5] 2023-06-23T01:23:47.522 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAi9JRk+xf5BxAA8BivwofdKlZTEPsozWLgTA== 2023-06-23T01:23:47.522 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.6] 2023-06-23T01:23:47.522 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAi9JRkjsbrEBAA/YOAXrf67tBQOnIhM8mL5g== 2023-06-23T01:23:47.522 INFO:teuthology.orchestra.run.smithi067.stdout:[osd.7] 2023-06-23T01:23:47.523 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAi9JRk56cCGhAAlzo9JTcMZh6cm831TK0/4g== 2023-06-23T01:23:47.523 INFO:teuthology.orchestra.run.smithi067.stdout:[client.0] 2023-06-23T01:23:47.523 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAb9JRkLqDeBRAAI7SQgvSm0iIZlXnP0Cfd1A== 2023-06-23T01:23:47.523 INFO:teuthology.orchestra.run.smithi067.stdout:[client.2] 2023-06-23T01:23:47.523 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAb9JRkINNzCxAAs6qURORdREx61QGuUoNUMQ== 2023-06-23T01:23:47.523 INFO:teuthology.orchestra.run.smithi067.stdout:[client.1] 2023-06-23T01:23:47.524 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAb9JRkKe+EDhAARtw3KZqHTpeMGfcDeUuRXw== 2023-06-23T01:23:47.524 INFO:teuthology.orchestra.run.smithi067.stdout:[client.3] 2023-06-23T01:23:47.524 INFO:teuthology.orchestra.run.smithi067.stdout: key = AQAb9JRkD8TgExAAN6dkJX5jsdIfG9XUCRrA+Q== 2023-06-23T01:23:47.539 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.y] 2023-06-23T01:23:47.540 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAa9JRkxjpFIxAAJfAa076whBzTR7UBFd/DEA== 2023-06-23T01:23:47.540 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.a] 2023-06-23T01:23:47.540 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAa9JRkUmSrKhAAj+j81xXc4pyIEb/FSzbxZg== 2023-06-23T01:23:47.540 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.c] 2023-06-23T01:23:47.540 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAa9JRkdUTBMRAAQHT0bf2rORbANYJYWUHLQQ== 2023-06-23T01:23:47.541 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.0] 2023-06-23T01:23:47.541 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAe9JRkKZOtGRAA571AepEUjHUQFgGjs6cxEQ== 2023-06-23T01:23:47.541 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.1] 2023-06-23T01:23:47.541 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAe9JRkouG9IBAAzSh7OwpoFejMHaduJ7S/Dw== 2023-06-23T01:23:47.541 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.2] 2023-06-23T01:23:47.541 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAe9JRkxqv3KRAAgZ7hzl9+KmMidcD5OAzHPw== 2023-06-23T01:23:47.541 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.3] 2023-06-23T01:23:47.542 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAe9JRkNYyENhAAcRB/FK4kxcIeU7Jb4nvE3g== 2023-06-23T01:23:47.542 INFO:teuthology.orchestra.run.smithi193.stdout:[mgr.x] 2023-06-23T01:23:47.542 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAa9JRkkvBwJxAA6xoW9R0SsDXcrMlRilTiLw== 2023-06-23T01:23:47.542 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.b] 2023-06-23T01:23:47.542 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAa9JRkjnZcNhAAEs+mQw+Igz2nS4r3YvDJ4A== 2023-06-23T01:23:47.542 INFO:teuthology.orchestra.run.smithi193.stdout:[mds.d] 2023-06-23T01:23:47.542 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAb9JRkffCPARAAQtAdBnWbfxHiN/G+9OQnyA== 2023-06-23T01:23:47.543 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.4] 2023-06-23T01:23:47.543 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAi9JRkxjMdARAA6YDqLhW7Q3AwdiDfnxBTrQ== 2023-06-23T01:23:47.543 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.5] 2023-06-23T01:23:47.543 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAi9JRk+xf5BxAA8BivwofdKlZTEPsozWLgTA== 2023-06-23T01:23:47.543 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.6] 2023-06-23T01:23:47.543 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAi9JRkjsbrEBAA/YOAXrf67tBQOnIhM8mL5g== 2023-06-23T01:23:47.543 INFO:teuthology.orchestra.run.smithi193.stdout:[osd.7] 2023-06-23T01:23:47.544 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAi9JRk56cCGhAAlzo9JTcMZh6cm831TK0/4g== 2023-06-23T01:23:47.544 INFO:teuthology.orchestra.run.smithi193.stdout:[client.0] 2023-06-23T01:23:47.544 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAb9JRkLqDeBRAAI7SQgvSm0iIZlXnP0Cfd1A== 2023-06-23T01:23:47.544 INFO:teuthology.orchestra.run.smithi193.stdout:[client.2] 2023-06-23T01:23:47.544 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAb9JRkINNzCxAAs6qURORdREx61QGuUoNUMQ== 2023-06-23T01:23:47.544 INFO:teuthology.orchestra.run.smithi193.stdout:[client.1] 2023-06-23T01:23:47.544 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAb9JRkKe+EDhAARtw3KZqHTpeMGfcDeUuRXw== 2023-06-23T01:23:47.544 INFO:teuthology.orchestra.run.smithi193.stdout:[client.3] 2023-06-23T01:23:47.545 INFO:teuthology.orchestra.run.smithi193.stdout: key = AQAb9JRkD8TgExAAN6dkJX5jsdIfG9XUCRrA+Q== 2023-06-23T01:23:47.546 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:23:47.568 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:23:47.633 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:23:47.635 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:23:47.687 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:23:47.690 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:23:47.743 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:47.746 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:47.793 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:47.796 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:47.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:47.845 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:47.893 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:47.895 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:47.942 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:23:47.945 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:23:47.992 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:23:47.994 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:23:48.050 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:23:48.054 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:23:48.105 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:48.108 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:48.155 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:48.158 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:48.205 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:48.210 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:48.257 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:48.260 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:23:48.307 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:23:48.310 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:23:48.361 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:23:48.364 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:23:48.414 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:23:48.416 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:23:48.472 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:23:48.475 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:23:48.561 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-06-23T01:23:48.562 DEBUG:teuthology.orchestra.run.smithi067:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-06-23T01:23:48.587 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:23:48.680 DEBUG:teuthology.orchestra.run.smithi067:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-06-23T01:23:48.705 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-06-23T01:23:48.729 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:23:48.826 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-06-23T01:23:48.850 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-06-23T01:23:48.916 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:23:49.008 DEBUG:teuthology.orchestra.run.smithi193:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-06-23T01:23:49.033 DEBUG:teuthology.orchestra.run.smithi067:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:49.035 DEBUG:teuthology.orchestra.run.smithi193:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:23:49.087 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-06-23T01:23:49.088 INFO:tasks.ceph.mon.a:Restarting daemon 2023-06-23T01:23:49.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-06-23T01:23:49.090 INFO:tasks.ceph.mon.a:Started 2023-06-23T01:23:49.091 INFO:tasks.ceph.mon.b:Restarting daemon 2023-06-23T01:23:49.091 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-06-23T01:23:49.130 INFO:tasks.ceph.mon.b:Started 2023-06-23T01:23:49.131 INFO:tasks.ceph.mon.c:Restarting daemon 2023-06-23T01:23:49.131 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-06-23T01:23:49.134 INFO:tasks.ceph.mon.c:Started 2023-06-23T01:23:49.135 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-06-23T01:23:49.135 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-06-23T01:23:49.135 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-06-23T01:23:49.137 INFO:tasks.ceph.mgr.y:Started 2023-06-23T01:23:49.138 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-06-23T01:23:49.138 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-06-23T01:23:49.140 INFO:tasks.ceph.mgr.x:Started 2023-06-23T01:23:49.140 INFO:tasks.ceph:Setting crush tunables to default 2023-06-23T01:23:49.141 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd crush tunables default 2023-06-23T01:23:49.264 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.261+0000 7f6a12ebd700 1 Processor -- start 2023-06-23T01:23:49.265 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.261+0000 7f6a12ebd700 1 -- start start 2023-06-23T01:23:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.261+0000 7f6a12ebd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.261+0000 7f6a12ebd700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c0c5c30 con 0x7f6a0c0cb8b0 2023-06-23T01:23:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.261+0000 7f6a12ebd700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c0c5f10 con 0x7f6a0c159eb0 2023-06-23T01:23:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a12ebd700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0c15e8f0 con 0x7f6a0c159ae0 2023-06-23T01:23:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a10c59700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a1145a700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f6a0c0cb8b0 0x7f6a0c15dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:47996/0 (socket says 172.21.15.67:47996) 2023-06-23T01:23:49.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a1145a700 1 -- 172.21.15.67:0/212200880 learned_addr learned my addr 172.21.15.67:0/212200880 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:23:49.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39574/0 (socket says 172.21.15.67:39574) 2023-06-23T01:23:49.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:23:49.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:49.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:49.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.262+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:23:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.463+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.463+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:23:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.463+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:49.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.463+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:49.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.463+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-23T01:23:49.866 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.864+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.864+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:23:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.864+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.864+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:49.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:49.864+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-23T01:23:50.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:50.665+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:50.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:50.665+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:23:50.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:50.665+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:50.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:50.665+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:50.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:50.665+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-23T01:23:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 msgr2=0x7f6a0c0c69f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-23T01:23:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0a7fc700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.67:3300/0 conn(0x7f6a0c159ae0 0x7f6a0c0c69f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 >> v1:172.21.15.193:6789/0 conn(0x7f6a0c159eb0 legacy=0x7f6a0c0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 >> v1:172.21.15.67:6789/0 conn(0x7f6a0c0cb8b0 legacy=0x7f6a0c15dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0a7fc700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:52.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0a7fc700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:52.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a0c15e8f0 con 0x7f69ec003710 2023-06-23T01:23:52.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ec008d60 con 0x7f69ec0061c0 2023-06-23T01:23:52.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:52.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:52.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:23:52.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:52.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:23:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0bfff700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-23T01:23:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0bfff700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:52.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.263+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:23:52.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:52.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:52.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-23T01:23:52.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:52.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a0bfff700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:23:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-23T01:23:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a0bfff700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:52.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.464+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-06-23T01:23:52.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:52.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:52.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:23:52.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:52.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:52.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a0bfff700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-23T01:23:52.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-23T01:23:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a0bfff700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:52.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:52.865+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-06-23T01:23:53.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:53.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:53.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a0bfff700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:23:53.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a0bfff700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:53.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:53.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-23T01:23:53.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-23T01:23:53.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:23:53.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:23:53.670 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:53.666+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-06-23T01:23:55.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-23T01:23:55.266 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0a7fc700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 msgr2=0x7f69ec005b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-23T01:23:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0a7fc700 1 --2- 172.21.15.67:0/212200880 >> v2:172.21.15.193:3300/0 conn(0x7f69ec003710 0x7f69ec005b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0a7fc700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a1145a700 1 -- 172.21.15.67:0/212200880 reap_dead start 2023-06-23T01:23:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a0c0c61f0 con 0x7f69ec00a880 2023-06-23T01:23:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ec00f520 con 0x7f69ec003b70 2023-06-23T01:23:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0a7fc700 1 -- 172.21.15.67:0/212200880 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c15e8f0 con 0x7f69ec0061c0 2023-06-23T01:23:55.267 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0b7fe700 1 -- 172.21.15.67:0/212200880 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356542176 0 0) 0x7f6a0c0c61f0 con 0x7f69ec00a880 2023-06-23T01:23:55.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.263+0000 7f6a0b7fe700 1 -- 172.21.15.67:0/212200880 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69f8003570 con 0x7f69ec00a880 2023-06-23T01:23:55.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.264+0000 7f6a0b7fe700 1 -- 172.21.15.67:0/212200880 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265853150 0 0) 0x7f69ec00f520 con 0x7f69ec003b70 2023-06-23T01:23:55.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.264+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v1:172.21.15.193:6789/0 conn(0x7f69ec003b70 legacy=0x7f69ec009160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.268 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.264+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 >> v1:172.21.15.67:6789/0 conn(0x7f69ec00a880 legacy=0x7f69ec00cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.264+0000 7f6a10c59700 1 -- 172.21.15.67:0/212200880 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ec010770 con 0x7f69ec0061c0 2023-06-23T01:23:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.264+0000 7f6a10c59700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f69f4017ab0 tx=0x7f69f4008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff7fe6c8ce156951 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.264+0000 7f6a0b7fe700 1 -- 172.21.15.67:0/212200880 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f401e070 con 0x7f69ec0061c0 2023-06-23T01:23:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.264+0000 7f6a0b7fe700 1 -- 172.21.15.67:0/212200880 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f40198d0 con 0x7f69ec0061c0 2023-06-23T01:23:55.269 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.264+0000 7f6a0b7fe700 1 -- 172.21.15.67:0/212200880 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f4019c50 con 0x7f69ec0061c0 2023-06-23T01:23:55.272 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.265+0000 7f6a12ebd700 1 -- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 msgr2=0x7f69ec008630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.265+0000 7f6a12ebd700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f69f4017ab0 tx=0x7f69f4008040 comp rx=0 tx=0).stop 2023-06-23T01:23:55.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.265+0000 7f6a12ebd700 1 -- 172.21.15.67:0/212200880 shutdown_connections 2023-06-23T01:23:55.273 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.265+0000 7f6a12ebd700 1 --2- 172.21.15.67:0/212200880 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec0061c0 0x7f69ec008630 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.265+0000 7f6a12ebd700 1 -- 172.21.15.67:0/212200880 >> 172.21.15.67:0/212200880 conn(0x7f6a0c063020 msgr2=0x7f6a0c063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:55.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.265+0000 7f6a12ebd700 1 -- 172.21.15.67:0/212200880 shutdown_connections 2023-06-23T01:23:55.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.265+0000 7f6a12ebd700 1 -- 172.21.15.67:0/212200880 wait complete. 2023-06-23T01:23:55.274 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.266+0000 7f6a12ebd700 1 Processor -- start 2023-06-23T01:23:55.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.266+0000 7f6a12ebd700 1 -- start start 2023-06-23T01:23:55.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a12ebd700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec003b70 0x7f6a0c06cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a12ebd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ec0061c0 0x7f6a0c14f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.275 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a12ebd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ec00a880 0x7f6a0c0b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a12ebd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c0b4980 con 0x7f69ec0061c0 2023-06-23T01:23:55.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a12ebd700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c06d670 con 0x7f69ec00a880 2023-06-23T01:23:55.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a12ebd700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0c06d340 con 0x7f69ec003b70 2023-06-23T01:23:55.276 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a0bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ec0061c0 0x7f6a0c14f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a0bfff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ec0061c0 0x7f6a0c14f710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:39692/0 (socket says 172.21.15.67:39692) 2023-06-23T01:23:55.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a0bfff700 1 -- 172.21.15.67:0/1602916935 learned_addr learned my addr 172.21.15.67:0/1602916935 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:23:55.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a10c59700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec003b70 0x7f6a0c06cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.277 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a1145a700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ec00a880 0x7f6a0c0b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a0bfff700 1 -- 172.21.15.67:0/1602916935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec003b70 msgr2=0x7f6a0c06cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec003b70 0x7f6a0c06cbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.278 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a0bfff700 1 -- 172.21.15.67:0/1602916935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ec00a880 msgr2=0x7f6a0c0b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ec00a880 0x7f6a0c0b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.267+0000 7f6a0bfff700 1 -- 172.21.15.67:0/1602916935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f400e040 con 0x7f69ec0061c0 2023-06-23T01:23:55.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.268+0000 7f6a0bfff700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ec0061c0 0x7f6a0c14f710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f69fc013650 tx=0x7f69fc009040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:55.279 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.268+0000 7f6a08ff9700 1 -- 172.21.15.67:0/1602916935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc006820 con 0x7f69ec0061c0 2023-06-23T01:23:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.268+0000 7f6a08ff9700 1 -- 172.21.15.67:0/1602916935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69fc0069a0 con 0x7f69ec0061c0 2023-06-23T01:23:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.268+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a0c06d120 con 0x7f69ec0061c0 2023-06-23T01:23:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.268+0000 7f6a08ff9700 1 -- 172.21.15.67:0/1602916935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69fc006d60 con 0x7f69ec0061c0 2023-06-23T01:23:55.280 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.268+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a0c1517d0 con 0x7f69ec0061c0 2023-06-23T01:23:55.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.268+0000 7f6a08ff9700 1 -- 172.21.15.67:0/1602916935 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f69fc007240 con 0x7f69ec0061c0 2023-06-23T01:23:55.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.268+0000 7f6a08ff9700 1 -- 172.21.15.67:0/1602916935 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f69fc016d30 con 0x7f69ec0061c0 2023-06-23T01:23:55.281 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.269+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a0c06d7e0 con 0x7f69ec0061c0 2023-06-23T01:23:55.282 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.271+0000 7f6a08ff9700 1 -- 172.21.15.67:0/1602916935 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f69fc023060 con 0x7f69ec0061c0 2023-06-23T01:23:55.351 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.348+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f6a0c05f060 con 0x7f69ec0061c0 2023-06-23T01:23:55.353 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.351+0000 7f6a08ff9700 1 -- 172.21.15.67:0/1602916935 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f69fc02d090 con 0x7f69ec0061c0 2023-06-23T01:23:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ec0061c0 msgr2=0x7f6a0c14f710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ec0061c0 0x7f6a0c14f710 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f69fc013650 tx=0x7f69fc009040 comp rx=0 tx=0).stop 2023-06-23T01:23:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 shutdown_connections 2023-06-23T01:23:55.355 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f69ec00a880 0x7f6a0c0b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f69ec003b70 0x7f6a0c06cbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 --2- 172.21.15.67:0/1602916935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f69ec0061c0 0x7f6a0c14f710 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 >> 172.21.15.67:0/1602916935 conn(0x7f6a0c063020 msgr2=0x7f6a0c068bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 shutdown_connections 2023-06-23T01:23:55.356 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:55.352+0000 7f6a12ebd700 1 -- 172.21.15.67:0/1602916935 wait complete. 2023-06-23T01:23:55.357 INFO:teuthology.orchestra.run.smithi067.stderr:adjusted tunables profile to default 2023-06-23T01:23:55.368 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-06-23T01:23:55.368 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:55.369 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-06-23T01:23:55.397 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:55.398 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-06-23T01:23:55.464 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:55.465 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-06-23T01:23:55.531 DEBUG:teuthology.orchestra.run.smithi067:> set -ex 2023-06-23T01:23:55.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-06-23T01:23:55.598 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:55.598 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-06-23T01:23:55.623 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:55.623 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-06-23T01:23:55.691 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:55.692 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-06-23T01:23:55.755 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2023-06-23T01:23:55.756 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-06-23T01:23:55.810 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:55.807+0000 7fdc8d22c180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:23:55.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 0 2023-06-23T01:23:55.949 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:55.946+0000 7fdc8d22c180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:23:55.978 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.976+0000 7f4893686700 1 Processor -- start 2023-06-23T01:23:55.978 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.976+0000 7f4893686700 1 -- start start 2023-06-23T01:23:55.980 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4893686700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f488c159c30 0x7f488c0acab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.980 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4893686700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f488c0697f0 0x7f488c0c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.981 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4893686700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f488c0c9680 con 0x7f488c15a000 2023-06-23T01:23:55.981 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4893686700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c0ad5c0 con 0x7f488c159c30 2023-06-23T01:23:55.981 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4893686700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f488c0ad8a0 con 0x7f488c0697f0 2023-06-23T01:23:55.982 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891c23700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f488c0697f0 0x7f488c0c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.982 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891c23700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f488c0697f0 0x7f488c0c9b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45706/0 (socket says 172.21.15.193:45706) 2023-06-23T01:23:55.982 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891c23700 1 -- 172.21.15.193:0/3002713210 learned_addr learned my addr 172.21.15.193:0/3002713210 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:55.982 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891422700 1 --2- 172.21.15.193:0/3002713210 >> v2:172.21.15.67:3300/0 conn(0x7f488c159c30 0x7f488c0acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.982 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891c23700 1 -- 172.21.15.193:0/3002713210 >> v2:172.21.15.67:3300/0 conn(0x7f488c159c30 msgr2=0x7f488c0acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:55.982 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891c23700 1 --2- 172.21.15.193:0/3002713210 >> v2:172.21.15.67:3300/0 conn(0x7f488c159c30 0x7f488c0acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.982 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891c23700 1 -- 172.21.15.193:0/3002713210 >> v1:172.21.15.67:6789/0 conn(0x7f488c15a000 legacy=0x7f488c0690e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.986 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891c23700 1 -- 172.21.15.193:0/3002713210 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f488c0adb80 con 0x7f488c0697f0 2023-06-23T01:23:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.977+0000 7f4891c23700 1 --2- 172.21.15.193:0/3002713210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c0697f0 0x7f488c0c9b10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f488c0af440 tx=0x7f4884009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=126528c29519a43e server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4883fff700 1 -- 172.21.15.193:0/3002713210 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488400e070 con 0x7f488c0697f0 2023-06-23T01:23:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4883fff700 1 -- 172.21.15.193:0/3002713210 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f488400bab0 con 0x7f488c0697f0 2023-06-23T01:23:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4883fff700 1 -- 172.21.15.193:0/3002713210 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48840092e0 con 0x7f488c0697f0 2023-06-23T01:23:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4893686700 1 -- 172.21.15.193:0/3002713210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c0697f0 msgr2=0x7f488c0c9b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4893686700 1 --2- 172.21.15.193:0/3002713210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c0697f0 0x7f488c0c9b10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f488c0af440 tx=0x7f4884009840 comp rx=0 tx=0).stop 2023-06-23T01:23:55.987 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4893686700 1 -- 172.21.15.193:0/3002713210 shutdown_connections 2023-06-23T01:23:55.988 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4893686700 1 --2- 172.21.15.193:0/3002713210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c0697f0 0x7f488c0c9b10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.988 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4893686700 1 --2- 172.21.15.193:0/3002713210 >> v2:172.21.15.67:3300/0 conn(0x7f488c159c30 0x7f488c0acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.988 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4893686700 1 -- 172.21.15.193:0/3002713210 >> 172.21.15.193:0/3002713210 conn(0x7f488c0cd6f0 msgr2=0x7f488c0679c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:55.988 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4893686700 1 -- 172.21.15.193:0/3002713210 shutdown_connections 2023-06-23T01:23:55.988 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.978+0000 7f4893686700 1 -- 172.21.15.193:0/3002713210 wait complete. 2023-06-23T01:23:55.988 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.979+0000 7f4893686700 1 Processor -- start 2023-06-23T01:23:55.988 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.979+0000 7f4893686700 1 -- start start 2023-06-23T01:23:55.989 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4893686700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0697f0 0x7f488c1486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.989 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4893686700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c159c30 0x7f488c15ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.989 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4893686700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f488c15a000 0x7f488c068b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:55.989 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4893686700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c15a480 con 0x7f488c0697f0 2023-06-23T01:23:55.989 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4893686700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f488c15f320 con 0x7f488c159c30 2023-06-23T01:23:55.989 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4893686700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488c15a720 con 0x7f488c15a000 2023-06-23T01:23:55.990 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4891c23700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f488c15a000 0x7f488c068b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.990 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4890c21700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c159c30 0x7f488c15ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.990 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4891c23700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f488c15a000 0x7f488c068b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:55440/0 (socket says 172.21.15.193:55440) 2023-06-23T01:23:55.990 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4890c21700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c159c30 0x7f488c15ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45708/0 (socket says 172.21.15.193:45708) 2023-06-23T01:23:55.990 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4891c23700 1 -- 172.21.15.193:0/3084051328 learned_addr learned my addr 172.21.15.193:0/3084051328 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:55.990 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.980+0000 7f4891422700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0697f0 0x7f488c1486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:55.990 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4891c23700 1 -- 172.21.15.193:0/3084051328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c159c30 msgr2=0x7f488c15ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.991 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4891c23700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c159c30 0x7f488c15ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.991 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4891c23700 1 -- 172.21.15.193:0/3084051328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0697f0 msgr2=0x7f488c1486f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:55.991 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4891c23700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0697f0 0x7f488c1486f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:55.991 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4891c23700 1 -- 172.21.15.193:0/3084051328 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4884009ae0 con 0x7f488c15a000 2023-06-23T01:23:55.991 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4890c21700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c159c30 0x7f488c15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:55.991 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4891c23700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f488c15a000 0x7f488c068b70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4884012000 tx=0x7f4884018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:55.992 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f48817fa700 1 -- 172.21.15.193:0/3084051328 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48840032f0 con 0x7f488c15a000 2023-06-23T01:23:55.992 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f48817fa700 1 -- 172.21.15.193:0/3084051328 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4884008ce0 con 0x7f488c15a000 2023-06-23T01:23:55.992 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488c15aa00 con 0x7f488c15a000 2023-06-23T01:23:55.992 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f48817fa700 1 -- 172.21.15.193:0/3084051328 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f488401d670 con 0x7f488c15a000 2023-06-23T01:23:55.992 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488c148c30 con 0x7f488c15a000 2023-06-23T01:23:55.992 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f48817fa700 1 -- 172.21.15.193:0/3084051328 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f4884003040 con 0x7f488c15a000 2023-06-23T01:23:55.992 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.981+0000 7f48817fa700 1 -- 172.21.15.193:0/3084051328 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f48840277a0 con 0x7f488c15a000 2023-06-23T01:23:55.993 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.982+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f488c05f060 con 0x7f488c15a000 2023-06-23T01:23:55.993 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:55.984+0000 7f48817fa700 1 -- 172.21.15.193:0/3084051328 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f488402c060 con 0x7f488c15a000 2023-06-23T01:23:56.029 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:56.026+0000 7fdc8d22c180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:23:56.070 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.067+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 0} v 0) v1 -- 0x7f488c15af20 con 0x7f488c15a000 2023-06-23T01:23:56.074 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.071+0000 7f48817fa700 1 -- 172.21.15.193:0/3084051328 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f488c15af20 con 0x7f488c15a000 2023-06-23T01:23:56.075 INFO:teuthology.orchestra.run.smithi193.stdout:0 2023-06-23T01:23:56.077 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.072+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f488c15a000 msgr2=0x7f488c068b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.077 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.072+0000 7f4893686700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f488c15a000 0x7f488c068b70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4884012000 tx=0x7f4884018000 comp rx=0 tx=0).stop 2023-06-23T01:23:56.078 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.072+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 shutdown_connections 2023-06-23T01:23:56.078 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.072+0000 7f4893686700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f488c15a000 0x7f488c068b70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.078 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.072+0000 7f4893686700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f488c0697f0 0x7f488c1486f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.072+0000 7f4893686700 1 --2- 172.21.15.193:0/3084051328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f488c159c30 0x7f488c15ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.072+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 >> 172.21.15.193:0/3084051328 conn(0x7f488c0cd6f0 msgr2=0x7f488c0cbd40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:56.079 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.072+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 shutdown_connections 2023-06-23T01:23:56.080 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.073+0000 7f4893686700 1 -- 172.21.15.193:0/3084051328 wait complete. 2023-06-23T01:23:56.087 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 1 2023-06-23T01:23:56.205 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.203+0000 7f3f52981700 1 Processor -- start 2023-06-23T01:23:56.206 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.203+0000 7f3f52981700 1 -- start start 2023-06-23T01:23:56.207 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f52981700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3f4c159ae0 0x7f3f4c0c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.208 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f52981700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3f4c0cb8b0 0x7f3f4c15dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.208 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f52981700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4c0c5c30 con 0x7f3f4c159eb0 2023-06-23T01:23:56.208 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f52981700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c15e8f0 con 0x7f3f4c159ae0 2023-06-23T01:23:56.208 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f52981700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c0c5f10 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.208 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f50f1e700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3f4c0cb8b0 0x7f3f4c15dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.208 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f4b7fe700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f3f4c159eb0 0x7f3f4c0cb1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:60286/0 (socket says 172.21.15.193:60286) 2023-06-23T01:23:56.209 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f4b7fe700 1 -- 172.21.15.193:0/3368407605 learned_addr learned my addr 172.21.15.193:0/3368407605 (peer_addr_for_me v1:172.21.15.193:0/0) 2023-06-23T01:23:56.209 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f4bfff700 1 --2- 172.21.15.193:0/3368407605 >> v2:172.21.15.67:3300/0 conn(0x7f3f4c159ae0 0x7f3f4c0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.209 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f50f1e700 1 -- 172.21.15.193:0/3368407605 >> v2:172.21.15.67:3300/0 conn(0x7f3f4c159ae0 msgr2=0x7f3f4c0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:56.209 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f50f1e700 1 --2- 172.21.15.193:0/3368407605 >> v2:172.21.15.67:3300/0 conn(0x7f3f4c159ae0 0x7f3f4c0c69f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.209 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f50f1e700 1 -- 172.21.15.193:0/3368407605 >> v1:172.21.15.193:6789/0 conn(0x7f3f4c159eb0 legacy=0x7f3f4c0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.209 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f50f1e700 1 -- 172.21.15.193:0/3368407605 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c0c61f0 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.209 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f4affd700 1 -- 172.21.15.193:0/3368407605 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691080273 0 0) 0x7f3f4c0c5c30 con 0x7f3f4c159eb0 2023-06-23T01:23:56.209 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f50f1e700 1 --2- 172.21.15.193:0/3368407605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c15dba0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0ccc90 tx=0x7f3f44009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f063a862fbfd1e55 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:56.210 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f4affd700 1 -- 172.21.15.193:0/3368407605 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4400e070 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.210 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f4affd700 1 -- 172.21.15.193:0/3368407605 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4400bb60 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.210 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.204+0000 7f3f4affd700 1 -- 172.21.15.193:0/3368407605 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f44012430 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.210 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.205+0000 7f3f52981700 1 -- 172.21.15.193:0/3368407605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 msgr2=0x7f3f4c15dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.210 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.205+0000 7f3f52981700 1 --2- 172.21.15.193:0/3368407605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c15dba0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c0ccc90 tx=0x7f3f44009780 comp rx=0 tx=0).stop 2023-06-23T01:23:56.210 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.205+0000 7f3f52981700 1 -- 172.21.15.193:0/3368407605 shutdown_connections 2023-06-23T01:23:56.210 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.205+0000 7f3f52981700 1 --2- 172.21.15.193:0/3368407605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c15dba0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.211 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.205+0000 7f3f52981700 1 --2- 172.21.15.193:0/3368407605 >> v2:172.21.15.67:3300/0 conn(0x7f3f4c159ae0 0x7f3f4c0c69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.211 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.205+0000 7f3f52981700 1 -- 172.21.15.193:0/3368407605 >> 172.21.15.193:0/3368407605 conn(0x7f3f4c063020 msgr2=0x7f3f4c063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:56.211 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.205+0000 7f3f52981700 1 -- 172.21.15.193:0/3368407605 shutdown_connections 2023-06-23T01:23:56.211 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.205+0000 7f3f52981700 1 -- 172.21.15.193:0/3368407605 wait complete. 2023-06-23T01:23:56.213 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.206+0000 7f3f52981700 1 Processor -- start 2023-06-23T01:23:56.214 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.206+0000 7f3f52981700 1 -- start start 2023-06-23T01:23:56.214 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f52981700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c06eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.214 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f52981700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f4c159ae0 0x7f3f4c0b3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.214 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f52981700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3f4c159eb0 0x7f3f4c155d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.214 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f52981700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c06eff0 con 0x7f3f4c159ae0 2023-06-23T01:23:56.215 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f52981700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c06e620 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.215 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f52981700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c06da80 con 0x7f3f4c159eb0 2023-06-23T01:23:56.215 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c06eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.215 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c06eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45730/0 (socket says 172.21.15.193:45730) 2023-06-23T01:23:56.215 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 -- 172.21.15.193:0/748541242 learned_addr learned my addr 172.21.15.193:0/748541242 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:56.215 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4b7fe700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f4c159ae0 0x7f3f4c0b3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.216 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f50f1e700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3f4c159eb0 0x7f3f4c155d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.216 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 -- 172.21.15.193:0/748541242 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3f4c159eb0 msgr2=0x7f3f4c155d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.216 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3f4c159eb0 0x7f3f4c155d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.216 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 -- 172.21.15.193:0/748541242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f4c159ae0 msgr2=0x7f3f4c0b3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.216 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f4c159ae0 0x7f3f4c0b3ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.216 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 -- 172.21.15.193:0/748541242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f44009a20 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.216 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.207+0000 7f3f4bfff700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c06eab0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000ecb0 tx=0x7f3f400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:56.216 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.208+0000 7f3f33fff700 1 -- 172.21.15.193:0/748541242 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4000c850 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.217 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.208+0000 7f3f33fff700 1 -- 172.21.15.193:0/748541242 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4000c9b0 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.217 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.208+0000 7f3f33fff700 1 -- 172.21.15.193:0/748541242 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4000cd50 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.217 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.208+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c06dcf0 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.217 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.208+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c06e2b0 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.217 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.208+0000 7f3f33fff700 1 -- 172.21.15.193:0/748541242 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3f40005c80 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.217 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.208+0000 7f3f33fff700 1 -- 172.21.15.193:0/748541242 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f3f40006020 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.217 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.209+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c05f060 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.218 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.211+0000 7f3f33fff700 1 -- 172.21.15.193:0/748541242 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f3f40021060 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.222 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:56.219+0000 7fdc8d22c180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:23:56.327 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.325+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 1} v 0) v1 -- 0x7f3f4c0c79d0 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.332 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.328+0000 7f3f33fff700 1 -- 172.21.15.193:0/748541242 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3f40017070 con 0x7f3f4c0cb8b0 2023-06-23T01:23:56.332 INFO:teuthology.orchestra.run.smithi193.stdout:1 2023-06-23T01:23:56.333 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 msgr2=0x7f3f4c06eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.333 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c06eab0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000ecb0 tx=0x7f3f400091a0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.333 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 shutdown_connections 2023-06-23T01:23:56.333 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3f4c159eb0 0x7f3f4c155d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.334 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f4c0cb8b0 0x7f3f4c06eab0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.334 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 --2- 172.21.15.193:0/748541242 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3f4c159ae0 0x7f3f4c0b3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.334 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 >> 172.21.15.193:0/748541242 conn(0x7f3f4c063020 msgr2=0x7f3f4c154e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:56.334 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 shutdown_connections 2023-06-23T01:23:56.334 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.329+0000 7f3f52981700 1 -- 172.21.15.193:0/748541242 wait complete. 2023-06-23T01:23:56.347 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 2 2023-06-23T01:23:56.403 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:56.400+0000 7fdc8d22c180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:23:56.464 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.461+0000 7f855fbc7700 1 Processor -- start 2023-06-23T01:23:56.464 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.461+0000 7f855fbc7700 1 -- start start 2023-06-23T01:23:56.466 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855fbc7700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8558156860 0x7f85580b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.466 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855fbc7700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f85580cac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.467 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855fbc7700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85580ca780 con 0x7f8558156c30 2023-06-23T01:23:56.467 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855fbc7700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f855815b4a0 con 0x7f8558156860 2023-06-23T01:23:56.467 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855fbc7700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85580c9b70 con 0x7f855815ad60 2023-06-23T01:23:56.467 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855e164700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f85580cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.467 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855d963700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f8558156860 0x7f85580b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.467 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855e164700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f85580cac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:55458/0 (socket says 172.21.15.193:55458) 2023-06-23T01:23:56.467 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.462+0000 7f855e164700 1 -- 172.21.15.193:0/3203980472 learned_addr learned my addr 172.21.15.193:0/3203980472 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:56.468 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855d963700 1 -- 172.21.15.193:0/3203980472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 msgr2=0x7f85580cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:56.468 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855d963700 1 --2- 172.21.15.193:0/3203980472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f85580cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.468 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855d963700 1 -- 172.21.15.193:0/3203980472 >> v1:172.21.15.67:6789/0 conn(0x7f8558156c30 legacy=0x7f855815a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.468 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855d963700 1 -- 172.21.15.193:0/3203980472 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85580c9e50 con 0x7f8558156860 2023-06-23T01:23:56.468 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855c961700 1 -- 172.21.15.193:0/3203980472 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902334540 0 0) 0x7f85580ca780 con 0x7f8558156c30 2023-06-23T01:23:56.468 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855e164700 1 --2- 172.21.15.193:0/3203980472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f85580cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:56.468 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855d963700 1 --2- 172.21.15.193:0/3203980472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f85580b1030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f85540098e0 tx=0x7f85540091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1838ff63ab2a5968 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:56.469 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855c961700 1 -- 172.21.15.193:0/3203980472 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855400b640 con 0x7f8558156860 2023-06-23T01:23:56.469 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855c961700 1 -- 172.21.15.193:0/3203980472 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855400b7a0 con 0x7f8558156860 2023-06-23T01:23:56.469 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855c961700 1 -- 172.21.15.193:0/3203980472 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855400bb40 con 0x7f8558156860 2023-06-23T01:23:56.469 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855fbc7700 1 -- 172.21.15.193:0/3203980472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 msgr2=0x7f85580b1030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.469 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855fbc7700 1 --2- 172.21.15.193:0/3203980472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f85580b1030 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f85540098e0 tx=0x7f85540091a0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.469 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855fbc7700 1 -- 172.21.15.193:0/3203980472 shutdown_connections 2023-06-23T01:23:56.469 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855fbc7700 1 --2- 172.21.15.193:0/3203980472 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f85580cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.469 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855fbc7700 1 --2- 172.21.15.193:0/3203980472 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f85580b1030 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.470 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.463+0000 7f855fbc7700 1 -- 172.21.15.193:0/3203980472 >> 172.21.15.193:0/3203980472 conn(0x7f85580cdb30 msgr2=0x7f8558067420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:56.470 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.464+0000 7f855fbc7700 1 -- 172.21.15.193:0/3203980472 shutdown_connections 2023-06-23T01:23:56.470 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.464+0000 7f855fbc7700 1 -- 172.21.15.193:0/3203980472 wait complete. 2023-06-23T01:23:56.472 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855fbc7700 1 Processor -- start 2023-06-23T01:23:56.472 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855fbc7700 1 -- start start 2023-06-23T01:23:56.472 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855fbc7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f8558150600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.472 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855fbc7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8558156c30 0x7f85580c1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.473 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855fbc7700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f8558154dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.473 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855fbc7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85580c8a50 con 0x7f8558156c30 2023-06-23T01:23:56.473 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855fbc7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8558165e40 con 0x7f8558156860 2023-06-23T01:23:56.473 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855fbc7700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8558166120 con 0x7f855815ad60 2023-06-23T01:23:56.473 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855d963700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f8558150600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.474 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855e164700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f8558154dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.474 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855d963700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f8558150600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45748/0 (socket says 172.21.15.193:45748) 2023-06-23T01:23:56.474 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855d963700 1 -- 172.21.15.193:0/1160706526 learned_addr learned my addr 172.21.15.193:0/1160706526 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:56.474 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.465+0000 7f855d162700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8558156c30 0x7f85580c1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.474 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855e164700 1 -- 172.21.15.193:0/1160706526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 msgr2=0x7f8558150600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.474 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855e164700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f8558150600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.474 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855e164700 1 -- 172.21.15.193:0/1160706526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8558156c30 msgr2=0x7f85580c1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.475 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855e164700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8558156c30 0x7f85580c1e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.475 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855e164700 1 -- 172.21.15.193:0/1160706526 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8550009ae0 con 0x7f855815ad60 2023-06-23T01:23:56.475 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855d963700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f8558150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:56.475 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855e164700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f8558154dd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8550009e70 tx=0x7f855000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:56.476 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f854dffb700 1 -- 172.21.15.193:0/1160706526 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8550018070 con 0x7f855815ad60 2023-06-23T01:23:56.476 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f854dffb700 1 -- 172.21.15.193:0/1160706526 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855000f040 con 0x7f855815ad60 2023-06-23T01:23:56.476 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8554009420 con 0x7f855815ad60 2023-06-23T01:23:56.476 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f854dffb700 1 -- 172.21.15.193:0/1160706526 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f855000cb00 con 0x7f855815ad60 2023-06-23T01:23:56.476 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f854dffb700 1 -- 172.21.15.193:0/1160706526 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f8550003020 con 0x7f855815ad60 2023-06-23T01:23:56.476 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85581666e0 con 0x7f855815ad60 2023-06-23T01:23:56.476 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.466+0000 7f854dffb700 1 -- 172.21.15.193:0/1160706526 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f8550014480 con 0x7f855815ad60 2023-06-23T01:23:56.477 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.467+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8558166a50 con 0x7f855815ad60 2023-06-23T01:23:56.477 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.469+0000 7f854dffb700 1 -- 172.21.15.193:0/1160706526 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f8558166a50 con 0x7f855815ad60 2023-06-23T01:23:56.484 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:56.481+0000 7fdc8d22c180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:23:56.554 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.551+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 2} v 0) v1 -- 0x7f85580712e0 con 0x7f855815ad60 2023-06-23T01:23:56.558 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.555+0000 7f854dffb700 1 -- 172.21.15.193:0/1160706526 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8550014640 con 0x7f855815ad60 2023-06-23T01:23:56.558 INFO:teuthology.orchestra.run.smithi193.stdout:2 2023-06-23T01:23:56.559 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 msgr2=0x7f8558154dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.559 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f8558154dd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8550009e70 tx=0x7f855000b040 comp rx=0 tx=0).stop 2023-06-23T01:23:56.559 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 shutdown_connections 2023-06-23T01:23:56.560 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f855815ad60 0x7f8558154dd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.560 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8558156860 0x7f8558150600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.560 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 --2- 172.21.15.193:0/1160706526 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f8558156c30 0x7f85580c1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.560 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 >> 172.21.15.193:0/1160706526 conn(0x7f85580cdb30 msgr2=0x7f8558068270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:56.560 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 shutdown_connections 2023-06-23T01:23:56.560 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.556+0000 7f855fbc7700 1 -- 172.21.15.193:0/1160706526 wait complete. 2023-06-23T01:23:56.574 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 3 2023-06-23T01:23:56.625 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:56.622+0000 7fdc8d22c180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:23:56.626 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:56.623+0000 7f83299ed180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:23:56.690 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.688+0000 7f3e86dcd700 1 Processor -- start 2023-06-23T01:23:56.691 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.688+0000 7f3e86dcd700 1 -- start start 2023-06-23T01:23:56.696 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e86dcd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f3e80159eb0 0x7f3e800cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.696 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e86dcd700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e800cb6e0 0x7f3e8015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e86dcd700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e800cbb00 con 0x7f3e80159ae0 2023-06-23T01:23:56.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e86dcd700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e8015e8f0 con 0x7f3e80159eb0 2023-06-23T01:23:56.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e86dcd700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e800c5c30 con 0x7f3e800cb6e0 2023-06-23T01:23:56.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e8536a700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e800cb6e0 0x7f3e8015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e8536a700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e800cb6e0 0x7f3e8015dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:55468/0 (socket says 172.21.15.193:55468) 2023-06-23T01:23:56.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e8536a700 1 -- 172.21.15.193:0/3718633549 learned_addr learned my addr 172.21.15.193:0/3718633549 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:56.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e7ffff700 1 --2- 172.21.15.193:0/3718633549 >> v2:172.21.15.67:3300/0 conn(0x7f3e80159eb0 0x7f3e800cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e8536a700 1 -- 172.21.15.193:0/3718633549 >> v2:172.21.15.67:3300/0 conn(0x7f3e80159eb0 msgr2=0x7f3e800cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e8536a700 1 --2- 172.21.15.193:0/3718633549 >> v2:172.21.15.67:3300/0 conn(0x7f3e80159eb0 0x7f3e800cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e8536a700 1 -- 172.21.15.193:0/3718633549 >> v1:172.21.15.67:6789/0 conn(0x7f3e80159ae0 legacy=0x7f3e800c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e8536a700 1 -- 172.21.15.193:0/3718633549 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e800c5f10 con 0x7f3e800cb6e0 2023-06-23T01:23:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e7ffff700 1 --2- 172.21.15.193:0/3718633549 >> v2:172.21.15.67:3300/0 conn(0x7f3e80159eb0 0x7f3e800cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e8536a700 1 --2- 172.21.15.193:0/3718633549 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e800cb6e0 0x7f3e8015dba0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009f20 tx=0x7f3e78009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b00c5e2337d7eeb server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:56.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.689+0000 7f3e7f7fe700 1 -- 172.21.15.193:0/3718633549 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e7800e070 con 0x7f3e800cb6e0 2023-06-23T01:23:56.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e7f7fe700 1 -- 172.21.15.193:0/3718633549 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e7800b9d0 con 0x7f3e800cb6e0 2023-06-23T01:23:56.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e7f7fe700 1 -- 172.21.15.193:0/3718633549 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e780092e0 con 0x7f3e800cb6e0 2023-06-23T01:23:56.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e86dcd700 1 -- 172.21.15.193:0/3718633549 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e800cb6e0 msgr2=0x7f3e8015dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e86dcd700 1 --2- 172.21.15.193:0/3718633549 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e800cb6e0 0x7f3e8015dba0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f3e78009f20 tx=0x7f3e78009840 comp rx=0 tx=0).stop 2023-06-23T01:23:56.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e86dcd700 1 -- 172.21.15.193:0/3718633549 shutdown_connections 2023-06-23T01:23:56.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e86dcd700 1 --2- 172.21.15.193:0/3718633549 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e800cb6e0 0x7f3e8015dba0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e86dcd700 1 --2- 172.21.15.193:0/3718633549 >> v2:172.21.15.67:3300/0 conn(0x7f3e80159eb0 0x7f3e800cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e86dcd700 1 -- 172.21.15.193:0/3718633549 >> 172.21.15.193:0/3718633549 conn(0x7f3e80063020 msgr2=0x7f3e80063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:56.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e86dcd700 1 -- 172.21.15.193:0/3718633549 shutdown_connections 2023-06-23T01:23:56.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.690+0000 7f3e86dcd700 1 -- 172.21.15.193:0/3718633549 wait complete. 2023-06-23T01:23:56.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.691+0000 7f3e86dcd700 1 Processor -- start 2023-06-23T01:23:56.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.691+0000 7f3e86dcd700 1 -- start start 2023-06-23T01:23:56.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e86dcd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e800cb6e0 0x7f3e80169b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e86dcd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80159ae0 0x7f3e80142d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e86dcd700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e80159eb0 0x7f3e801432b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.701 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e86dcd700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e800d0720 con 0x7f3e80159ae0 2023-06-23T01:23:56.701 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e86dcd700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e800d0a00 con 0x7f3e800cb6e0 2023-06-23T01:23:56.701 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e86dcd700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e8016aca0 con 0x7f3e80159eb0 2023-06-23T01:23:56.701 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e84b69700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e800cb6e0 0x7f3e80169b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.701 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e8536a700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e80159eb0 0x7f3e801432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e84b69700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e800cb6e0 0x7f3e80169b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45750/0 (socket says 172.21.15.193:45750) 2023-06-23T01:23:56.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e84b69700 1 -- 172.21.15.193:0/1252681005 learned_addr learned my addr 172.21.15.193:0/1252681005 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:56.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.692+0000 7f3e7ffff700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80159ae0 0x7f3e80142d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e8536a700 1 -- 172.21.15.193:0/1252681005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e800cb6e0 msgr2=0x7f3e80169b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e8536a700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e800cb6e0 0x7f3e80169b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e8536a700 1 -- 172.21.15.193:0/1252681005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80159ae0 msgr2=0x7f3e80142d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e8536a700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80159ae0 0x7f3e80142d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e8536a700 1 -- 172.21.15.193:0/1252681005 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6c009420 con 0x7f3e80159eb0 2023-06-23T01:23:56.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e7ffff700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80159ae0 0x7f3e80142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:56.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e8536a700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e80159eb0 0x7f3e801432b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3e78015000 tx=0x7f3e78018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:56.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e7cff9700 1 -- 172.21.15.193:0/1252681005 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78008ad0 con 0x7f3e80159eb0 2023-06-23T01:23:56.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e7cff9700 1 -- 172.21.15.193:0/1252681005 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e78008c30 con 0x7f3e80159eb0 2023-06-23T01:23:56.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e78009ae0 con 0x7f3e80159eb0 2023-06-23T01:23:56.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e7cff9700 1 -- 172.21.15.193:0/1252681005 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e78025460 con 0x7f3e80159eb0 2023-06-23T01:23:56.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e8016a950 con 0x7f3e80159eb0 2023-06-23T01:23:56.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e7cff9700 1 -- 172.21.15.193:0/1252681005 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3e78003040 con 0x7f3e80159eb0 2023-06-23T01:23:56.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.693+0000 7f3e7cff9700 1 -- 172.21.15.193:0/1252681005 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f3e78025cc0 con 0x7f3e80159eb0 2023-06-23T01:23:56.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.694+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e8016b2f0 con 0x7f3e80159eb0 2023-06-23T01:23:56.705 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.696+0000 7f3e7cff9700 1 -- 172.21.15.193:0/1252681005 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f3e7802a020 con 0x7f3e80159eb0 2023-06-23T01:23:56.724 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:56.722+0000 7fdc8d22c180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:23:56.763 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:56.761+0000 7f83299ed180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:23:56.781 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.778+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 3} v 0) v1 -- 0x7f3e8005f060 con 0x7f3e80159eb0 2023-06-23T01:23:56.785 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.782+0000 7f3e7cff9700 1 -- 172.21.15.193:0/1252681005 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3e78020070 con 0x7f3e80159eb0 2023-06-23T01:23:56.785 INFO:teuthology.orchestra.run.smithi193.stdout:3 2023-06-23T01:23:56.786 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e80159eb0 msgr2=0x7f3e801432b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.786 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e80159eb0 0x7f3e801432b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f3e78015000 tx=0x7f3e78018000 comp rx=0 tx=0).stop 2023-06-23T01:23:56.786 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 shutdown_connections 2023-06-23T01:23:56.786 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3e80159eb0 0x7f3e801432b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.787 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e800cb6e0 0x7f3e80169b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.787 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 --2- 172.21.15.193:0/1252681005 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3e80159ae0 0x7f3e80142d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.787 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 >> 172.21.15.193:0/1252681005 conn(0x7f3e80063020 msgr2=0x7f3e8015db00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:56.787 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 shutdown_connections 2023-06-23T01:23:56.787 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.783+0000 7f3e86dcd700 1 -- 172.21.15.193:0/1252681005 wait complete. 2023-06-23T01:23:56.797 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 4 2023-06-23T01:23:56.844 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:56.841+0000 7f83299ed180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:23:56.915 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.912+0000 7f962c569700 1 Processor -- start 2023-06-23T01:23:56.915 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.912+0000 7f962c569700 1 -- start start 2023-06-23T01:23:56.923 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.913+0000 7f962c569700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f9624159eb0 0x7f96240cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.923 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.913+0000 7f962c569700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96240cbb00 con 0x7f96240cb6e0 2023-06-23T01:23:56.923 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.913+0000 7f962c569700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96240c5c30 con 0x7f9624159ae0 2023-06-23T01:23:56.923 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.913+0000 7f962c569700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962415e8f0 con 0x7f9624159eb0 2023-06-23T01:23:56.923 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.913+0000 7f962a305700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f9624159ae0 0x7f96240c69f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:60302/0 (socket says 172.21.15.193:60302) 2023-06-23T01:23:56.923 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.913+0000 7f962a305700 1 -- 172.21.15.193:0/2457650349 learned_addr learned my addr 172.21.15.193:0/2457650349 (peer_addr_for_me v1:172.21.15.193:0/0) 2023-06-23T01:23:56.924 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629b04700 1 --2- 172.21.15.193:0/2457650349 >> v2:172.21.15.67:3300/0 conn(0x7f9624159eb0 0x7f96240cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.924 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248921421 0 0) 0x7f96240c5c30 con 0x7f9624159ae0 2023-06-23T01:23:56.924 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9610003570 con 0x7f9624159ae0 2023-06-23T01:23:56.925 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 310570601 0 0) 0x7f9610003570 con 0x7f9624159ae0 2023-06-23T01:23:56.925 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 >> v2:172.21.15.67:3300/0 conn(0x7f9624159eb0 msgr2=0x7f96240cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:56.925 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 --2- 172.21.15.193:0/2457650349 >> v2:172.21.15.67:3300/0 conn(0x7f9624159eb0 0x7f96240cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.925 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 >> v1:172.21.15.67:6789/0 conn(0x7f96240cb6e0 legacy=0x7f962415dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.925 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96240c5f10 con 0x7f9624159ae0 2023-06-23T01:23:56.925 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 918208491 0 0) 0x7f9614001470 con 0x7f9624159ae0 2023-06-23T01:23:56.925 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 >> v1:172.21.15.193:6789/0 conn(0x7f9624159ae0 legacy=0x7f96240c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.925 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003a60 0x7f9610003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.926 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610004810 0x7f9610004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.926 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96100051c0 0x7f961000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.926 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f962415e8f0 con 0x7f9610003a60 2023-06-23T01:23:56.926 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f961000c0e0 con 0x7f96100051c0 2023-06-23T01:23:56.926 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f961000c2a0 con 0x7f9610004810 2023-06-23T01:23:56.926 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629b04700 1 --2- 172.21.15.193:0/2457650349 >> v2:172.21.15.67:3300/0 conn(0x7f9624159eb0 0x7f96240cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:56.926 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f962a305700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003a60 0x7f9610003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.926 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f962ab06700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96100051c0 0x7f961000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.927 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f9629b04700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610004810 0x7f9610004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.927 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f962a305700 1 -- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610004810 msgr2=0x7f9610004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:56.927 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f962a305700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610004810 0x7f9610004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.927 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f962a305700 1 -- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96100051c0 msgr2=0x7f961000ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:56.928 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.914+0000 7f962a305700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96100051c0 0x7f961000ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.928 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962a305700 1 -- 172.21.15.193:0/2457650349 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9610003570 con 0x7f9610003a60 2023-06-23T01:23:56.928 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962ab06700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96100051c0 0x7f961000ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:56.928 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962ab06700 1 -- 172.21.15.193:0/2457650349 reap_dead start 2023-06-23T01:23:56.928 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962a305700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003a60 0x7f9610003e30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f961400c3c0 tx=0x7f9614001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3d0741d85d3c86 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:56.928 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961400cc90 con 0x7f9610003a60 2023-06-23T01:23:56.928 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f961400cdf0 con 0x7f9610003a60 2023-06-23T01:23:56.929 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f9629303700 1 -- 172.21.15.193:0/2457650349 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614005b70 con 0x7f9610003a60 2023-06-23T01:23:56.929 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962c569700 1 -- 172.21.15.193:0/2457650349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003a60 msgr2=0x7f9610003e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.929 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962c569700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003a60 0x7f9610003e30 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f961400c3c0 tx=0x7f9614001e70 comp rx=0 tx=0).stop 2023-06-23T01:23:56.929 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962c569700 1 -- 172.21.15.193:0/2457650349 shutdown_connections 2023-06-23T01:23:56.929 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962c569700 1 --2- 172.21.15.193:0/2457650349 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f9610003a60 0x7f9610003e30 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.929 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.915+0000 7f962c569700 1 -- 172.21.15.193:0/2457650349 >> 172.21.15.193:0/2457650349 conn(0x7f9624063020 msgr2=0x7f9624063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:56.929 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.916+0000 7f962c569700 1 -- 172.21.15.193:0/2457650349 shutdown_connections 2023-06-23T01:23:56.930 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.916+0000 7f962c569700 1 -- 172.21.15.193:0/2457650349 wait complete. 2023-06-23T01:23:56.930 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962c569700 1 Processor -- start 2023-06-23T01:23:56.930 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962c569700 1 -- start start 2023-06-23T01:23:56.930 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962c569700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610003a60 0x7f9624152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.930 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962c569700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96240b5980 0x7f96240cd4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.930 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962c569700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962416a3d0 0x7f962416a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:56.930 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962c569700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9624169cc0 con 0x7f962416a3d0 2023-06-23T01:23:56.930 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962c569700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96240b6190 con 0x7f96240b5980 2023-06-23T01:23:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962c569700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96240b6470 con 0x7f9610003a60 2023-06-23T01:23:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962a305700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610003a60 0x7f9624152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f9629b04700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96240b5980 0x7f96240cd4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962a305700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610003a60 0x7f9624152be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:55492/0 (socket says 172.21.15.193:55492) 2023-06-23T01:23:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962a305700 1 -- 172.21.15.193:0/2646586509 learned_addr learned my addr 172.21.15.193:0/2646586509 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.917+0000 7f962ab06700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962416a3d0 0x7f962416a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f962a305700 1 -- 172.21.15.193:0/2646586509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96240b5980 msgr2=0x7f96240cd4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.931 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f962a305700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96240b5980 0x7f96240cd4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.932 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f962a305700 1 -- 172.21.15.193:0/2646586509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962416a3d0 msgr2=0x7f962416a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:56.932 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f962a305700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962416a3d0 0x7f962416a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:56.932 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f962a305700 1 -- 172.21.15.193:0/2646586509 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96240b6750 con 0x7f9610003a60 2023-06-23T01:23:56.932 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f9629b04700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96240b5980 0x7f96240cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:56.932 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f962a305700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610003a60 0x7f9624152be0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f961000c0b0 tx=0x7f9614015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:56.932 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f961a7fc700 1 -- 172.21.15.193:0/2646586509 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f961400cc90 con 0x7f9610003a60 2023-06-23T01:23:56.932 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f961a7fc700 1 -- 172.21.15.193:0/2646586509 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96140070d0 con 0x7f9610003a60 2023-06-23T01:23:56.933 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f961a7fc700 1 -- 172.21.15.193:0/2646586509 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9614007490 con 0x7f9610003a60 2023-06-23T01:23:56.933 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9624169eb0 con 0x7f9610003a60 2023-06-23T01:23:56.933 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96240b5df0 con 0x7f9610003a60 2023-06-23T01:23:56.933 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f961a7fc700 1 -- 172.21.15.193:0/2646586509 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f96140286b0 con 0x7f9610003a60 2023-06-23T01:23:56.933 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.918+0000 7f961a7fc700 1 -- 172.21.15.193:0/2646586509 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f9614007ae0 con 0x7f9610003a60 2023-06-23T01:23:56.933 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.919+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f962416a220 con 0x7f9610003a60 2023-06-23T01:23:56.934 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:56.921+0000 7f961a7fc700 1 -- 172.21.15.193:0/2646586509 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f961403f6e0 con 0x7f9610003a60 2023-06-23T01:23:57.028 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.025+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 4} v 0) v1 -- 0x7f962405f060 con 0x7f9610003a60 2023-06-23T01:23:57.032 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.029+0000 7f961a7fc700 1 -- 172.21.15.193:0/2646586509 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f961402e070 con 0x7f9610003a60 2023-06-23T01:23:57.032 INFO:teuthology.orchestra.run.smithi193.stdout:4 2023-06-23T01:23:57.033 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610003a60 msgr2=0x7f9624152be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.033 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610003a60 0x7f9624152be0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f961000c0b0 tx=0x7f9614015000 comp rx=0 tx=0).stop 2023-06-23T01:23:57.033 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 shutdown_connections 2023-06-23T01:23:57.033 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f962416a3d0 0x7f962416a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.034 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f9610003a60 0x7f9624152be0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.034 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 --2- 172.21.15.193:0/2646586509 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96240b5980 0x7f96240cd4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.034 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 >> 172.21.15.193:0/2646586509 conn(0x7f9624063020 msgr2=0x7f96241535d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:57.034 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 shutdown_connections 2023-06-23T01:23:57.034 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.030+0000 7f962c569700 1 -- 172.21.15.193:0/2646586509 wait complete. 2023-06-23T01:23:57.044 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 5 2023-06-23T01:23:57.054 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:57.051+0000 7f83299ed180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:23:57.160 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.157+0000 7f88d309f700 1 Processor -- start 2023-06-23T01:23:57.160 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.157+0000 7f88d309f700 1 -- start start 2023-06-23T01:23:57.166 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.158+0000 7f88d309f700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f88cc159eb0 0x7f88cc0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.166 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.158+0000 7f88d309f700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc0cbb00 con 0x7f88cc0cb6e0 2023-06-23T01:23:57.166 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.158+0000 7f88d309f700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88cc0c5c30 con 0x7f88cc159ae0 2023-06-23T01:23:57.167 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.158+0000 7f88d309f700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88cc15e8f0 con 0x7f88cc159eb0 2023-06-23T01:23:57.167 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.158+0000 7f88d0e3b700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f88cc159ae0 0x7f88cc0c69f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:60304/0 (socket says 172.21.15.193:60304) 2023-06-23T01:23:57.167 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.158+0000 7f88d0e3b700 1 -- 172.21.15.193:0/1352589922 learned_addr learned my addr 172.21.15.193:0/1352589922 (peer_addr_for_me v1:172.21.15.193:0/0) 2023-06-23T01:23:57.167 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.158+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2279991940 0 0) 0x7f88cc0c5c30 con 0x7f88cc159ae0 2023-06-23T01:23:57.167 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cbfff700 1 --2- 172.21.15.193:0/1352589922 >> v2:172.21.15.67:3300/0 conn(0x7f88cc159eb0 0x7f88cc0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.167 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88bc003570 con 0x7f88cc159ae0 2023-06-23T01:23:57.167 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 <== mon.1 v1:172.21.15.193:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3573670523 0 0) 0x7f88bc003570 con 0x7f88cc159ae0 2023-06-23T01:23:57.168 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 >> v2:172.21.15.67:3300/0 conn(0x7f88cc159eb0 msgr2=0x7f88cc0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:57.168 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 --2- 172.21.15.193:0/1352589922 >> v2:172.21.15.67:3300/0 conn(0x7f88cc159eb0 0x7f88cc0cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.168 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 >> v1:172.21.15.67:6789/0 conn(0x7f88cc0cb6e0 legacy=0x7f88cc15dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.168 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 --> v1:172.21.15.193:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc0c5f10 con 0x7f88cc159ae0 2023-06-23T01:23:57.168 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 <== mon.1 v1:172.21.15.193:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 918208491 0 0) 0x7f88c0001470 con 0x7f88cc159ae0 2023-06-23T01:23:57.168 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 >> v1:172.21.15.193:6789/0 conn(0x7f88cc159ae0 legacy=0x7f88cc0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.168 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc003a60 0x7f88bc003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88bc004810 0x7f88bc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88bc0051c0 0x7f88bc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc15e8f0 con 0x7f88bc003a60 2023-06-23T01:23:57.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc00c0e0 con 0x7f88bc0051c0 2023-06-23T01:23:57.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc00c2a0 con 0x7f88bc004810 2023-06-23T01:23:57.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d163c700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88bc0051c0 0x7f88bc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d0e3b700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc003a60 0x7f88bc003e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.169 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88cbfff700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88bc004810 0x7f88bc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.170 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d163c700 1 -- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88bc004810 msgr2=0x7f88bc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:57.170 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d163c700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88bc004810 0x7f88bc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.170 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d163c700 1 -- 172.21.15.193:0/1352589922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc003a60 msgr2=0x7f88bc003e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:57.170 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d163c700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc003a60 0x7f88bc003e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.170 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d163c700 1 -- 172.21.15.193:0/1352589922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc003570 con 0x7f88bc0051c0 2023-06-23T01:23:57.170 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d0e3b700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc003a60 0x7f88bc003e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:57.170 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d163c700 1 -- 172.21.15.193:0/1352589922 reap_dead start 2023-06-23T01:23:57.170 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.159+0000 7f88d163c700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88bc0051c0 0x7f88bc00ba60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f88c40020a0 tx=0x7f88c40016d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3f91a629e8224aa5 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:57.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c40019e0 con 0x7f88bc0051c0 2023-06-23T01:23:57.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c400c8b0 con 0x7f88bc0051c0 2023-06-23T01:23:57.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88cb7fe700 1 -- 172.21.15.193:0/1352589922 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c400cc50 con 0x7f88bc0051c0 2023-06-23T01:23:57.171 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88d309f700 1 -- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88bc0051c0 msgr2=0x7f88bc00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88d309f700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88bc0051c0 0x7f88bc00ba60 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f88c40020a0 tx=0x7f88c40016d0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88d309f700 1 -- 172.21.15.193:0/1352589922 shutdown_connections 2023-06-23T01:23:57.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88d309f700 1 --2- 172.21.15.193:0/1352589922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88bc0051c0 0x7f88bc00ba60 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88d309f700 1 -- 172.21.15.193:0/1352589922 >> 172.21.15.193:0/1352589922 conn(0x7f88cc063020 msgr2=0x7f88cc063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:57.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88d309f700 1 -- 172.21.15.193:0/1352589922 shutdown_connections 2023-06-23T01:23:57.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.160+0000 7f88d309f700 1 -- 172.21.15.193:0/1352589922 wait complete. 2023-06-23T01:23:57.172 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.161+0000 7f88d309f700 1 Processor -- start 2023-06-23T01:23:57.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.161+0000 7f88d309f700 1 -- start start 2023-06-23T01:23:57.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d309f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc0051c0 0x7f88cc1512b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d309f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88cc1517f0 0x7f88cc0b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d309f700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88cc0b4050 0x7f88cc1579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d309f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc150e20 con 0x7f88bc0051c0 2023-06-23T01:23:57.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d309f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc150210 con 0x7f88cc1517f0 2023-06-23T01:23:57.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d309f700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88cc1504f0 con 0x7f88cc0b4050 2023-06-23T01:23:57.173 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d163c700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88cc0b4050 0x7f88cc1579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.174 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88cbfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88cc1517f0 0x7f88cc0b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.174 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d163c700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88cc0b4050 0x7f88cc1579f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:55516/0 (socket says 172.21.15.193:55516) 2023-06-23T01:23:57.174 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88cbfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88cc1517f0 0x7f88cc0b3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45796/0 (socket says 172.21.15.193:45796) 2023-06-23T01:23:57.174 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d163c700 1 -- 172.21.15.193:0/3511002363 learned_addr learned my addr 172.21.15.193:0/3511002363 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:57.174 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d0e3b700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc0051c0 0x7f88cc1512b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.174 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d163c700 1 -- 172.21.15.193:0/3511002363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88cc1517f0 msgr2=0x7f88cc0b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.174 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.162+0000 7f88d163c700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88cc1517f0 0x7f88cc0b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.175 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88d163c700 1 -- 172.21.15.193:0/3511002363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc0051c0 msgr2=0x7f88cc1512b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.175 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88d163c700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc0051c0 0x7f88cc1512b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.175 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88d163c700 1 -- 172.21.15.193:0/3511002363 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88cc1507d0 con 0x7f88cc0b4050 2023-06-23T01:23:57.175 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88cbfff700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88cc1517f0 0x7f88cc0b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:57.175 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88d163c700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88cc0b4050 0x7f88cc1579f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0b6de0 tx=0x7f88c4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:57.175 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88c8ff9700 1 -- 172.21.15.193:0/3511002363 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c40019e0 con 0x7f88cc0b4050 2023-06-23T01:23:57.175 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88c8ff9700 1 -- 172.21.15.193:0/3511002363 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c400c6d0 con 0x7f88cc0b4050 2023-06-23T01:23:57.175 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88c8ff9700 1 -- 172.21.15.193:0/3511002363 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88c4007a40 con 0x7f88cc0b4050 2023-06-23T01:23:57.176 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88cc150ab0 con 0x7f88cc0b4050 2023-06-23T01:23:57.176 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88cc1580f0 con 0x7f88cc0b4050 2023-06-23T01:23:57.176 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88c8ff9700 1 -- 172.21.15.193:0/3511002363 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f88c401d8b0 con 0x7f88cc0b4050 2023-06-23T01:23:57.176 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.163+0000 7f88c8ff9700 1 -- 172.21.15.193:0/3511002363 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f88c4027a20 con 0x7f88cc0b4050 2023-06-23T01:23:57.176 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.164+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88cc05f060 con 0x7f88cc0b4050 2023-06-23T01:23:57.177 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.166+0000 7f88c8ff9700 1 -- 172.21.15.193:0/3511002363 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f88c4022020 con 0x7f88cc0b4050 2023-06-23T01:23:57.243 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:57.241+0000 7f83299ed180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:23:57.264 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.261+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 5} v 0) v1 -- 0x7f88cc158310 con 0x7f88cc0b4050 2023-06-23T01:23:57.268 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.265+0000 7f88c8ff9700 1 -- 172.21.15.193:0/3511002363 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f88c401dd90 con 0x7f88cc0b4050 2023-06-23T01:23:57.268 INFO:teuthology.orchestra.run.smithi193.stdout:5 2023-06-23T01:23:57.269 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88cc0b4050 msgr2=0x7f88cc1579f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.269 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88cc0b4050 0x7f88cc1579f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0b6de0 tx=0x7f88c4016000 comp rx=0 tx=0).stop 2023-06-23T01:23:57.269 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 shutdown_connections 2023-06-23T01:23:57.269 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f88cc0b4050 0x7f88cc1579f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.270 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f88bc0051c0 0x7f88cc1512b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.270 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 --2- 172.21.15.193:0/3511002363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f88cc1517f0 0x7f88cc0b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.270 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 >> 172.21.15.193:0/3511002363 conn(0x7f88cc063020 msgr2=0x7f88cc15db00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:57.270 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 shutdown_connections 2023-06-23T01:23:57.270 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.266+0000 7f88d309f700 1 -- 172.21.15.193:0/3511002363 wait complete. 2023-06-23T01:23:57.280 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 6 2023-06-23T01:23:57.323 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:57.321+0000 7f83299ed180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:23:57.439 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.436+0000 7fa6a79c7700 1 Processor -- start 2023-06-23T01:23:57.439 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.436+0000 7fa6a79c7700 1 -- start start 2023-06-23T01:23:57.441 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a79c7700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0156e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.441 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a79c7700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fa6a0069bd0 0x7fa6a00b0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.441 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a79c7700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a00b1a90 con 0x7fa6a006d9e0 2023-06-23T01:23:57.442 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a79c7700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6a00b0660 con 0x7fa6a0069bd0 2023-06-23T01:23:57.442 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a79c7700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00ce270 con 0x7fa6a0156a40 2023-06-23T01:23:57.442 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a5763700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0156e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.442 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a5f64700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fa6a006d9e0 0x7fa6a0068900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:60314/0 (socket says 172.21.15.193:60314) 2023-06-23T01:23:57.442 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a5f64700 1 -- 172.21.15.193:0/820956687 learned_addr learned my addr 172.21.15.193:0/820956687 (peer_addr_for_me v1:172.21.15.193:0/0) 2023-06-23T01:23:57.443 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.437+0000 7fa6a4f62700 1 --2- 172.21.15.193:0/820956687 >> v2:172.21.15.67:3300/0 conn(0x7fa6a0069bd0 0x7fa6a00b0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.443 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa697fff700 1 -- 172.21.15.193:0/820956687 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021278063 0 0) 0x7fa6a00b1a90 con 0x7fa6a006d9e0 2023-06-23T01:23:57.443 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa6a5763700 1 -- 172.21.15.193:0/820956687 >> v2:172.21.15.67:3300/0 conn(0x7fa6a0069bd0 msgr2=0x7fa6a00b0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:57.443 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa6a5763700 1 --2- 172.21.15.193:0/820956687 >> v2:172.21.15.67:3300/0 conn(0x7fa6a0069bd0 0x7fa6a00b0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.444 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa6a5763700 1 -- 172.21.15.193:0/820956687 >> v1:172.21.15.193:6789/0 conn(0x7fa6a006d9e0 legacy=0x7fa6a0068900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.444 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa6a5763700 1 -- 172.21.15.193:0/820956687 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00b1460 con 0x7fa6a0156a40 2023-06-23T01:23:57.444 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa6a5763700 1 --2- 172.21.15.193:0/820956687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0156e10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa6900098e0 tx=0x7fa6900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bea763eec48900d7 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:57.444 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa697fff700 1 -- 172.21.15.193:0/820956687 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69000b640 con 0x7fa6a0156a40 2023-06-23T01:23:57.444 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa697fff700 1 -- 172.21.15.193:0/820956687 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69000b7a0 con 0x7fa6a0156a40 2023-06-23T01:23:57.445 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.438+0000 7fa697fff700 1 -- 172.21.15.193:0/820956687 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa69000bb40 con 0x7fa6a0156a40 2023-06-23T01:23:57.445 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.439+0000 7fa6a79c7700 1 -- 172.21.15.193:0/820956687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 msgr2=0x7fa6a0156e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.445 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.439+0000 7fa6a79c7700 1 --2- 172.21.15.193:0/820956687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0156e10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa6900098e0 tx=0x7fa6900091a0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.445 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.439+0000 7fa6a79c7700 1 -- 172.21.15.193:0/820956687 shutdown_connections 2023-06-23T01:23:57.445 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.439+0000 7fa6a79c7700 1 --2- 172.21.15.193:0/820956687 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0156e10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.445 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.439+0000 7fa6a79c7700 1 --2- 172.21.15.193:0/820956687 >> v2:172.21.15.67:3300/0 conn(0x7fa6a0069bd0 0x7fa6a00b0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.445 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.439+0000 7fa6a79c7700 1 -- 172.21.15.193:0/820956687 >> 172.21.15.193:0/820956687 conn(0x7fa6a0067ef0 msgr2=0x7fa6a00cd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:57.446 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.439+0000 7fa6a79c7700 1 -- 172.21.15.193:0/820956687 shutdown_connections 2023-06-23T01:23:57.446 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.439+0000 7fa6a79c7700 1 -- 172.21.15.193:0/820956687 wait complete. 2023-06-23T01:23:57.447 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.440+0000 7fa6a79c7700 1 Processor -- start 2023-06-23T01:23:57.448 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.440+0000 7fa6a79c7700 1 -- start start 2023-06-23T01:23:57.448 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a79c7700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6a0069bd0 0x7fa6a014b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.448 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a79c7700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a006d9e0 0x7fa6a01588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.448 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a79c7700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0063020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.449 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a79c7700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a015a4d0 con 0x7fa6a006d9e0 2023-06-23T01:23:57.449 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a79c7700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a00c90f0 con 0x7fa6a0069bd0 2023-06-23T01:23:57.449 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a79c7700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6a0158df0 con 0x7fa6a0156a40 2023-06-23T01:23:57.449 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6a0069bd0 0x7fa6a014b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.449 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5f64700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0063020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.449 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6a0069bd0 0x7fa6a014b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45810/0 (socket says 172.21.15.193:45810) 2023-06-23T01:23:57.450 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5f64700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0063020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.193:55544/0 (socket says 172.21.15.193:55544) 2023-06-23T01:23:57.450 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 -- 172.21.15.193:0/929321028 learned_addr learned my addr 172.21.15.193:0/929321028 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:57.450 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a4f62700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a006d9e0 0x7fa6a01588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.450 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 -- 172.21.15.193:0/929321028 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 msgr2=0x7fa6a0063020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.450 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0063020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.450 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 -- 172.21.15.193:0/929321028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a006d9e0 msgr2=0x7fa6a01588b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.450 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a006d9e0 0x7fa6a01588b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.450 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 -- 172.21.15.193:0/929321028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa690009420 con 0x7fa6a0069bd0 2023-06-23T01:23:57.451 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5f64700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0063020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:57.451 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a4f62700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a006d9e0 0x7fa6a01588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:23:57.452 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a5763700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6a0069bd0 0x7fa6a014b8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa6900093f0 tx=0x7fa690017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:57.452 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6957fa700 1 -- 172.21.15.193:0/929321028 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690020070 con 0x7fa6a0069bd0 2023-06-23T01:23:57.452 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6957fa700 1 -- 172.21.15.193:0/929321028 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa690000c90 con 0x7fa6a0069bd0 2023-06-23T01:23:57.452 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a0159850 con 0x7fa6a0069bd0 2023-06-23T01:23:57.452 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6957fa700 1 -- 172.21.15.193:0/929321028 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa690026420 con 0x7fa6a0069bd0 2023-06-23T01:23:57.453 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.441+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a0159e10 con 0x7fa6a0069bd0 2023-06-23T01:23:57.453 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.442+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a006ec40 con 0x7fa6a0069bd0 2023-06-23T01:23:57.453 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.443+0000 7fa6957fa700 1 -- 172.21.15.193:0/929321028 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa690024070 con 0x7fa6a0069bd0 2023-06-23T01:23:57.453 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.443+0000 7fa6957fa700 1 -- 172.21.15.193:0/929321028 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7fa690011a20 con 0x7fa6a0069bd0 2023-06-23T01:23:57.454 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.446+0000 7fa6957fa700 1 -- 172.21.15.193:0/929321028 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fa69002b070 con 0x7fa6a0069bd0 2023-06-23T01:23:57.469 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:57.466+0000 7f83299ed180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:23:57.532 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.529+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 6} v 0) v1 -- 0x7fa6a005f060 con 0x7fa6a0069bd0 2023-06-23T01:23:57.541 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.537+0000 7fa6957fa700 1 -- 172.21.15.193:0/929321028 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fa690015020 con 0x7fa6a0069bd0 2023-06-23T01:23:57.541 INFO:teuthology.orchestra.run.smithi193.stdout:6 2023-06-23T01:23:57.542 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6a0069bd0 msgr2=0x7fa6a014b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.542 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6a0069bd0 0x7fa6a014b8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fa6900093f0 tx=0x7fa690017000 comp rx=0 tx=0).stop 2023-06-23T01:23:57.542 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 shutdown_connections 2023-06-23T01:23:57.542 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fa6a0156a40 0x7fa6a0063020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.542 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6a0069bd0 0x7fa6a014b8e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.543 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 --2- 172.21.15.193:0/929321028 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fa6a006d9e0 0x7fa6a01588b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.543 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 >> 172.21.15.193:0/929321028 conn(0x7fa6a0067ef0 msgr2=0x7fa6a01544e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:57.543 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 shutdown_connections 2023-06-23T01:23:57.543 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.538+0000 7fa6a79c7700 1 -- 172.21.15.193:0/929321028 wait complete. 2023-06-23T01:23:57.556 DEBUG:teuthology.orchestra.run.smithi193:> sudo ceph --cluster ceph osd new 00000000-0000-0000-0000-000000000000 7 2023-06-23T01:23:57.577 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:57.571+0000 7f83299ed180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:23:57.693 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.690+0000 7fcfe6628700 1 Processor -- start 2023-06-23T01:23:57.693 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.690+0000 7fcfe6628700 1 -- start start 2023-06-23T01:23:57.695 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfe6628700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fcfe0159ae0 0x7fcfe00c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.696 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfe6628700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fcfe0159eb0 0x7fcfe00cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.696 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfe6628700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfe00cbb00 con 0x7fcfe00cb6e0 2023-06-23T01:23:57.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfe6628700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe015e8f0 con 0x7fcfe0159eb0 2023-06-23T01:23:57.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfe6628700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfe00c5c30 con 0x7fcfe0159ae0 2023-06-23T01:23:57.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfdffff700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fcfe0159ae0 0x7fcfe00c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.697 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfe4bc5700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7fcfe00cb6e0 0x7fcfe015dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.193:60330/0 (socket says 172.21.15.193:60330) 2023-06-23T01:23:57.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfdffff700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fcfe0159ae0 0x7fcfe00c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45822/0 (socket says 172.21.15.193:45822) 2023-06-23T01:23:57.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.691+0000 7fcfe4bc5700 1 -- 172.21.15.193:0/2323126782 learned_addr learned my addr 172.21.15.193:0/2323126782 (peer_addr_for_me v1:172.21.15.193:0/0) 2023-06-23T01:23:57.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdffff700 1 -- 172.21.15.193:0/2323126782 >> v2:172.21.15.67:3300/0 conn(0x7fcfe0159eb0 msgr2=0x7fcfe00cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:57.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdffff700 1 --2- 172.21.15.193:0/2323126782 >> v2:172.21.15.67:3300/0 conn(0x7fcfe0159eb0 0x7fcfe00cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdffff700 1 -- 172.21.15.193:0/2323126782 >> v1:172.21.15.193:6789/0 conn(0x7fcfe00cb6e0 legacy=0x7fcfe015dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdeffd700 1 -- 172.21.15.193:0/2323126782 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1550856238 0 0) 0x7fcfe00cbb00 con 0x7fcfe00cb6e0 2023-06-23T01:23:57.698 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdffff700 1 -- 172.21.15.193:0/2323126782 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe00c5f10 con 0x7fcfe0159ae0 2023-06-23T01:23:57.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdffff700 1 --2- 172.21.15.193:0/2323126782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159ae0 0x7fcfe00c69f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcfe004ba30 tx=0x7fcfd40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c93c835422b55130 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:57.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdeffd700 1 -- 172.21.15.193:0/2323126782 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400b6b0 con 0x7fcfe0159ae0 2023-06-23T01:23:57.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdeffd700 1 -- 172.21.15.193:0/2323126782 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd400b810 con 0x7fcfe0159ae0 2023-06-23T01:23:57.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfdeffd700 1 -- 172.21.15.193:0/2323126782 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd400bbb0 con 0x7fcfe0159ae0 2023-06-23T01:23:57.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfe6628700 1 -- 172.21.15.193:0/2323126782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159ae0 msgr2=0x7fcfe00c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.699 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfe6628700 1 --2- 172.21.15.193:0/2323126782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159ae0 0x7fcfe00c69f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcfe004ba30 tx=0x7fcfd40091a0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.692+0000 7fcfe6628700 1 -- 172.21.15.193:0/2323126782 shutdown_connections 2023-06-23T01:23:57.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.693+0000 7fcfe6628700 1 --2- 172.21.15.193:0/2323126782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159ae0 0x7fcfe00c69f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.693+0000 7fcfe6628700 1 --2- 172.21.15.193:0/2323126782 >> v2:172.21.15.67:3300/0 conn(0x7fcfe0159eb0 0x7fcfe00cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.693+0000 7fcfe6628700 1 -- 172.21.15.193:0/2323126782 >> 172.21.15.193:0/2323126782 conn(0x7fcfe0063020 msgr2=0x7fcfe0063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:57.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.693+0000 7fcfe6628700 1 -- 172.21.15.193:0/2323126782 shutdown_connections 2023-06-23T01:23:57.700 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.693+0000 7fcfe6628700 1 -- 172.21.15.193:0/2323126782 wait complete. 2023-06-23T01:23:57.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.694+0000 7fcfe6628700 1 Processor -- start 2023-06-23T01:23:57.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.694+0000 7fcfe6628700 1 -- start start 2023-06-23T01:23:57.702 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.694+0000 7fcfe6628700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfe00cb6e0 0x7fcfe016d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.694+0000 7fcfe6628700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe0159ae0 0x7fcfe00c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.694+0000 7fcfe6628700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159eb0 0x7fcfe0157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:57.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.694+0000 7fcfe6628700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe015a440 con 0x7fcfe0159ae0 2023-06-23T01:23:57.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe6628700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe016df10 con 0x7fcfe0159eb0 2023-06-23T01:23:57.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe6628700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfe016e1f0 con 0x7fcfe00cb6e0 2023-06-23T01:23:57.703 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159eb0 0x7fcfe0157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfdffff700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfe00cb6e0 0x7fcfe016d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159eb0 0x7fcfe0157910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:45830/0 (socket says 172.21.15.193:45830) 2023-06-23T01:23:57.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 -- 172.21.15.193:0/3424198073 learned_addr learned my addr 172.21.15.193:0/3424198073 (peer_addr_for_me v2:172.21.15.193:0/0) 2023-06-23T01:23:57.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfdf7fe700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe0159ae0 0x7fcfe00c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:57.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 -- 172.21.15.193:0/3424198073 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfe00cb6e0 msgr2=0x7fcfe016d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfe00cb6e0 0x7fcfe016d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.704 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 -- 172.21.15.193:0/3424198073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe0159ae0 msgr2=0x7fcfe00c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.705 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe0159ae0 0x7fcfe00c9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.705 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 -- 172.21.15.193:0/3424198073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd4009420 con 0x7fcfe0159eb0 2023-06-23T01:23:57.705 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfdffff700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfe00cb6e0 0x7fcfe016d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:23:57.705 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe4bc5700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159eb0 0x7fcfe0157910 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800cbf0 tx=0x7fcfd80019b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:57.705 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfc7fff700 1 -- 172.21.15.193:0/3424198073 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd800a8b0 con 0x7fcfe0159eb0 2023-06-23T01:23:57.705 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfc7fff700 1 -- 172.21.15.193:0/3424198073 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd800aa10 con 0x7fcfe0159eb0 2023-06-23T01:23:57.705 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfe016e4d0 con 0x7fcfe0159eb0 2023-06-23T01:23:57.706 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfc7fff700 1 -- 172.21.15.193:0/3424198073 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfd800add0 con 0x7fcfe0159eb0 2023-06-23T01:23:57.706 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.695+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfe0157e50 con 0x7fcfe0159eb0 2023-06-23T01:23:57.706 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.697+0000 7fcfc7fff700 1 -- 172.21.15.193:0/3424198073 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcfd8007300 con 0x7fcfe0159eb0 2023-06-23T01:23:57.706 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.697+0000 7fcfc7fff700 1 -- 172.21.15.193:0/3424198073 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7fcfd801ba70 con 0x7fcfe0159eb0 2023-06-23T01:23:57.706 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.697+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfe015a5f0 con 0x7fcfe0159eb0 2023-06-23T01:23:57.706 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.699+0000 7fcfc7fff700 1 -- 172.21.15.193:0/3424198073 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fcfd8007920 con 0x7fcfe0159eb0 2023-06-23T01:23:57.783 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.780+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 7} v 0) v1 -- 0x7fcfe005f060 con 0x7fcfe0159eb0 2023-06-23T01:23:57.791 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.788+0000 7fcfc7fff700 1 -- 172.21.15.193:0/3424198073 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00000000-0000-0000-0000-000000000000", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcfd802a0b0 con 0x7fcfe0159eb0 2023-06-23T01:23:57.791 INFO:teuthology.orchestra.run.smithi193.stdout:7 2023-06-23T01:23:57.793 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.789+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159eb0 msgr2=0x7fcfe0157910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:57.793 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.789+0000 7fcfe6628700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159eb0 0x7fcfe0157910 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800cbf0 tx=0x7fcfd80019b0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.793 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.789+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 shutdown_connections 2023-06-23T01:23:57.793 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.789+0000 7fcfe6628700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcfe0159eb0 0x7fcfe0157910 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.793 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.789+0000 7fcfe6628700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fcfe00cb6e0 0x7fcfe016d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.794 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.789+0000 7fcfe6628700 1 --2- 172.21.15.193:0/3424198073 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fcfe0159ae0 0x7fcfe00c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:57.794 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.790+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 >> 172.21.15.193:0/3424198073 conn(0x7fcfe0063020 msgr2=0x7fcfe00cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:57.794 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.790+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 shutdown_connections 2023-06-23T01:23:57.794 INFO:teuthology.orchestra.run.smithi193.stderr:2023-06-23T01:23:57.790+0000 7fcfe6628700 1 -- 172.21.15.193:0/3424198073 wait complete. 2023-06-23T01:23:57.804 INFO:tasks.ceph.osd.0:Restarting daemon 2023-06-23T01:23:57.812 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-06-23T01:23:57.815 INFO:tasks.ceph.osd.0:Started 2023-06-23T01:23:57.815 INFO:tasks.ceph.osd.1:Restarting daemon 2023-06-23T01:23:57.816 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-06-23T01:23:57.818 INFO:tasks.ceph.osd.1:Started 2023-06-23T01:23:57.819 INFO:tasks.ceph.osd.2:Restarting daemon 2023-06-23T01:23:57.819 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-06-23T01:23:57.824 INFO:tasks.ceph.osd.2:Started 2023-06-23T01:23:57.825 INFO:tasks.ceph.osd.3:Restarting daemon 2023-06-23T01:23:57.825 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-06-23T01:23:57.830 INFO:tasks.ceph.osd.3:Started 2023-06-23T01:23:57.831 INFO:tasks.ceph.osd.4:Restarting daemon 2023-06-23T01:23:57.831 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-06-23T01:23:57.834 INFO:tasks.ceph.osd.4:Started 2023-06-23T01:23:57.834 INFO:tasks.ceph.osd.5:Restarting daemon 2023-06-23T01:23:57.834 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-06-23T01:23:57.836 INFO:tasks.ceph.osd.5:Started 2023-06-23T01:23:57.837 INFO:tasks.ceph.osd.6:Restarting daemon 2023-06-23T01:23:57.837 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-06-23T01:23:57.839 INFO:tasks.ceph.osd.6:Started 2023-06-23T01:23:57.840 INFO:tasks.ceph.osd.7:Restarting daemon 2023-06-23T01:23:57.840 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-06-23T01:23:57.843 INFO:tasks.ceph.osd.7:Started 2023-06-23T01:23:57.843 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:23:57.912 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-06-23T01:23:57.909+0000 7f9c8c12b540 -1 Falling back to public interface 2023-06-23T01:23:57.916 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-06-23T01:23:57.913+0000 7f9c6adc6540 -1 Falling back to public interface 2023-06-23T01:23:57.921 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-06-23T01:23:57.918+0000 7f47f7b49540 -1 Falling back to public interface 2023-06-23T01:23:57.925 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-06-23T01:23:57.923+0000 7f9c8c12b540 -1 osd.4 0 log_to_monitors true 2023-06-23T01:23:57.928 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-06-23T01:23:57.924+0000 7f5dafe80540 -1 Falling back to public interface 2023-06-23T01:23:57.933 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-06-23T01:23:57.931+0000 7f9c6adc6540 -1 osd.5 0 log_to_monitors true 2023-06-23T01:23:57.933 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-06-23T01:23:57.931+0000 7f2988d99540 -1 Falling back to public interface 2023-06-23T01:23:57.934 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-06-23T01:23:57.931+0000 7f6b2af10540 -1 Falling back to public interface 2023-06-23T01:23:57.936 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-06-23T01:23:57.933+0000 7f8639f1f540 -1 Falling back to public interface 2023-06-23T01:23:57.937 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-06-23T01:23:57.935+0000 7f47f7b49540 -1 osd.7 0 log_to_monitors true 2023-06-23T01:23:57.940 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-06-23T01:23:57.938+0000 7efcecf38540 -1 Falling back to public interface 2023-06-23T01:23:57.944 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-06-23T01:23:57.942+0000 7f5dafe80540 -1 osd.0 0 log_to_monitors true 2023-06-23T01:23:57.951 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-06-23T01:23:57.948+0000 7f8639f1f540 -1 osd.6 0 log_to_monitors true 2023-06-23T01:23:57.953 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-06-23T01:23:57.951+0000 7f2988d99540 -1 osd.2 0 log_to_monitors true 2023-06-23T01:23:57.955 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-06-23T01:23:57.953+0000 7f6b2af10540 -1 osd.1 0 log_to_monitors true 2023-06-23T01:23:57.958 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-06-23T01:23:57.956+0000 7efcecf38540 -1 osd.3 0 log_to_monitors true 2023-06-23T01:23:58.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.002+0000 7fca7dde3700 1 Processor -- start 2023-06-23T01:23:58.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.002+0000 7fca7dde3700 1 -- start start 2023-06-23T01:23:58.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.003+0000 7fca7dde3700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7fca78157bf0 0x7fca78156a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:58.011 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.003+0000 7fca7dde3700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca7806e190 con 0x7fca78156650 2023-06-23T01:23:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.003+0000 7fca7dde3700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca780b1d50 con 0x7fca7806e620 2023-06-23T01:23:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.003+0000 7fca7dde3700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca780b1fe0 con 0x7fca78157bf0 2023-06-23T01:23:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.003+0000 7fca777fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fca78156650 0x7fca780b1640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37228/0 (socket says 172.21.15.67:37228) 2023-06-23T01:23:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.003+0000 7fca777fe700 1 -- 172.21.15.67:0/3379647791 learned_addr learned my addr 172.21.15.67:0/3379647791 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:23:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca76ffd700 1 --2- 172.21.15.67:0/3379647791 >> v2:172.21.15.193:3300/0 conn(0x7fca78157bf0 0x7fca78156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:58.012 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562012295 0 0) 0x7fca7806e190 con 0x7fca78156650 2023-06-23T01:23:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca64003570 con 0x7fca78156650 2023-06-23T01:23:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3327100537 0 0) 0x7fca780b1d50 con 0x7fca7806e620 2023-06-23T01:23:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca7806e190 con 0x7fca7806e620 2023-06-23T01:23:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2736795960 0 0) 0x7fca64003570 con 0x7fca78156650 2023-06-23T01:23:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 >> v2:172.21.15.193:3300/0 conn(0x7fca78157bf0 msgr2=0x7fca78156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:58.013 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 --2- 172.21.15.67:0/3379647791 >> v2:172.21.15.193:3300/0 conn(0x7fca78157bf0 0x7fca78156a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 >> v1:172.21.15.193:6789/0 conn(0x7fca7806e620 legacy=0x7fca78067420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:58.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca7806db40 con 0x7fca78156650 2023-06-23T01:23:58.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 918208491 0 0) 0x7fca68001470 con 0x7fca78156650 2023-06-23T01:23:58.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 >> v1:172.21.15.67:6789/0 conn(0x7fca78156650 legacy=0x7fca780b1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:58.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca64003bf0 0x7fca64003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:58.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64004810 0x7fca64004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:58.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca640051c0 0x7fca6400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:58.014 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca780b1fe0 con 0x7fca64003bf0 2023-06-23T01:23:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6400c0b0 con 0x7fca64004810 2023-06-23T01:23:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca6400c270 con 0x7fca640051c0 2023-06-23T01:23:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca777fe700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca64003bf0 0x7fca64003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca76ffd700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64004810 0x7fca64004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca77fff700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca640051c0 0x7fca6400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca777fe700 1 -- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca640051c0 msgr2=0x7fca6400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.004+0000 7fca777fe700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca640051c0 0x7fca6400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.015 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca777fe700 1 -- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64004810 msgr2=0x7fca64004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca777fe700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64004810 0x7fca64004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca777fe700 1 -- 172.21.15.67:0/3379647791 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca64003570 con 0x7fca64003bf0 2023-06-23T01:23:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca77fff700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca640051c0 0x7fca6400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:23:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca77fff700 1 -- 172.21.15.67:0/3379647791 reap_dead start 2023-06-23T01:23:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca777fe700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca64003bf0 0x7fca64003fe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fca6800c2d0 tx=0x7fca68001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4eac71fefc2fc802 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca6800cba0 con 0x7fca64003bf0 2023-06-23T01:23:58.016 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca6800cd00 con 0x7fca64003bf0 2023-06-23T01:23:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca767fc700 1 -- 172.21.15.67:0/3379647791 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca68005b70 con 0x7fca64003bf0 2023-06-23T01:23:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca7dde3700 1 -- 172.21.15.67:0/3379647791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca64003bf0 msgr2=0x7fca64003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca7dde3700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca64003bf0 0x7fca64003fe0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fca6800c2d0 tx=0x7fca68001e10 comp rx=0 tx=0).stop 2023-06-23T01:23:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca7dde3700 1 -- 172.21.15.67:0/3379647791 shutdown_connections 2023-06-23T01:23:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca7dde3700 1 --2- 172.21.15.67:0/3379647791 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca64003bf0 0x7fca64003fe0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca7dde3700 1 -- 172.21.15.67:0/3379647791 >> 172.21.15.67:0/3379647791 conn(0x7fca780cd900 msgr2=0x7fca780cdd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:58.017 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca7dde3700 1 -- 172.21.15.67:0/3379647791 shutdown_connections 2023-06-23T01:23:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.005+0000 7fca7dde3700 1 -- 172.21.15.67:0/3379647791 wait complete. 2023-06-23T01:23:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.007+0000 7fca7dde3700 1 Processor -- start 2023-06-23T01:23:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.007+0000 7fca7dde3700 1 -- start start 2023-06-23T01:23:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.007+0000 7fca7dde3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64003bf0 0x7fca78147e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.007+0000 7fca7dde3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca78145170 0x7fca781485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.007+0000 7fca7dde3700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca78148b10 0x7fca7814d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.007+0000 7fca7dde3700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca78146da0 con 0x7fca78145170 2023-06-23T01:23:58.018 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.007+0000 7fca7dde3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca78147080 con 0x7fca64003bf0 2023-06-23T01:23:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.007+0000 7fca7dde3700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca78147360 con 0x7fca78148b10 2023-06-23T01:23:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca76ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca78145170 0x7fca781485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca77fff700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca78148b10 0x7fca7814d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca777fe700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64003bf0 0x7fca78147e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca76ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca78145170 0x7fca781485d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56614/0 (socket says 172.21.15.67:56614) 2023-06-23T01:23:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca76ffd700 1 -- 172.21.15.67:0/1053674428 learned_addr learned my addr 172.21.15.67:0/1053674428 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:23:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca77fff700 1 -- 172.21.15.67:0/1053674428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64003bf0 msgr2=0x7fca78147e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:58.019 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca77fff700 1 --2- 172.21.15.67:0/1053674428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64003bf0 0x7fca78147e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca77fff700 1 -- 172.21.15.67:0/1053674428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca78145170 msgr2=0x7fca781485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca77fff700 1 --2- 172.21.15.67:0/1053674428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca78145170 0x7fca781485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca77fff700 1 -- 172.21.15.67:0/1053674428 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca78147640 con 0x7fca78148b10 2023-06-23T01:23:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca77fff700 1 --2- 172.21.15.67:0/1053674428 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca78148b10 0x7fca7814d5e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fca7804c2c0 tx=0x7fca70009040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca6f7fe700 1 -- 172.21.15.67:0/1053674428 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca700032e0 con 0x7fca78148b10 2023-06-23T01:23:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca6f7fe700 1 -- 172.21.15.67:0/1053674428 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca70003440 con 0x7fca78148b10 2023-06-23T01:23:58.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca78147920 con 0x7fca78148b10 2023-06-23T01:23:58.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca6f7fe700 1 -- 172.21.15.67:0/1053674428 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70006df0 con 0x7fca78148b10 2023-06-23T01:23:58.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca7814db20 con 0x7fca78148b10 2023-06-23T01:23:58.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca6f7fe700 1 -- 172.21.15.67:0/1053674428 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fca70001b70 con 0x7fca78148b10 2023-06-23T01:23:58.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.008+0000 7fca6f7fe700 1 -- 172.21.15.67:0/1053674428 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fca70001e20 con 0x7fca78148b10 2023-06-23T01:23:58.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.009+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca78147ab0 con 0x7fca78148b10 2023-06-23T01:23:58.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.011+0000 7fca6f7fe700 1 -- 172.21.15.67:0/1053674428 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fca70019070 con 0x7fca78148b10 2023-06-23T01:23:58.080 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:58.078+0000 7fdc8d22c180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:23:58.098 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.096+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca781471c0 con 0x7fca78148b10 2023-06-23T01:23:58.099 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.097+0000 7fca6f7fe700 1 -- 172.21.15.67:0/1053674428 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fca781471c0 con 0x7fca78148b10 2023-06-23T01:23:58.099 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:23:58.099 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":10,"fsid":"0068cd93-d7dd-4502-a85d-c8368ea7e9d2","created":"2023-06-23T01:23:54.450197+0000","modified":"2023-06-23T01:23:57.782500+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:23:57.782500+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.69999998807907104,"backfillfull_ratio":0.60000002384185791,"nearfull_ratio":0.60000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:23:58.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca78148b10 msgr2=0x7fca7814d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:58.100 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 --2- 172.21.15.67:0/1053674428 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca78148b10 0x7fca7814d5e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fca7804c2c0 tx=0x7fca70009040 comp rx=0 tx=0).stop 2023-06-23T01:23:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 shutdown_connections 2023-06-23T01:23:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 --2- 172.21.15.67:0/1053674428 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fca78148b10 0x7fca7814d5e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 --2- 172.21.15.67:0/1053674428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fca64003bf0 0x7fca78147e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 --2- 172.21.15.67:0/1053674428 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fca78145170 0x7fca781485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 >> 172.21.15.67:0/1053674428 conn(0x7fca780cd900 msgr2=0x7fca7814bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:58.101 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 shutdown_connections 2023-06-23T01:23:58.102 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:23:58.098+0000 7fca7dde3700 1 -- 172.21.15.67:0/1053674428 wait complete. 2023-06-23T01:23:58.112 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-06-23T01:23:58.113 INFO:tasks.ceph:Waiting for OSDs to come up 2023-06-23T01:23:58.132 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:58.129+0000 7f83299ed180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:23:58.164 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:58.161+0000 7fdc8d22c180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:23:58.209 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:58.206+0000 7f83299ed180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:23:58.245 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:58.243+0000 7fdc8d22c180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:23:58.285 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:58.283+0000 7f83299ed180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:23:58.856 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:58.854+0000 7fdc8d22c180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:23:58.862 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:58.859+0000 7f83299ed180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:23:58.919 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:23:58.998 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:58.995+0000 7f83299ed180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:23:59.003 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:59.001+0000 7fdc8d22c180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:23:59.036 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.033+0000 7fd80ea42700 1 Processor -- start 2023-06-23T01:23:59.036 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.034+0000 7fd80ea42700 1 -- start start 2023-06-23T01:23:59.039 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.034+0000 7fd80ea42700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 0x7fd80815ac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:59.039 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.034+0000 7fd80ea42700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8080af7d0 0x7fd8080aa870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:59.039 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.034+0000 7fd80ea42700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd80806eee0 con 0x7fd8081542f0 2023-06-23T01:23:59.040 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.034+0000 7fd80ea42700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd80815b1c0 con 0x7fd8080af7d0 2023-06-23T01:23:59.040 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.034+0000 7fd80ea42700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80806f370 con 0x7fd8080adad0 2023-06-23T01:23:59.041 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd80cfdf700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7fd8080af7d0 0x7fd8080aa870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:59.041 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd807fff700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7fd8081542f0 0x7fd8080ad3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37244/0 (socket says 172.21.15.67:37244) 2023-06-23T01:23:59.041 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd807fff700 1 -- 172.21.15.67:0/187493648 learned_addr learned my addr 172.21.15.67:0/187493648 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:23:59.041 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd8077fe700 1 --2- 172.21.15.67:0/187493648 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 0x7fd80815ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:59.042 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd80cfdf700 1 -- 172.21.15.67:0/187493648 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 msgr2=0x7fd80815ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:23:59.042 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd80cfdf700 1 --2- 172.21.15.67:0/187493648 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 0x7fd80815ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.043 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd80cfdf700 1 -- 172.21.15.67:0/187493648 >> v1:172.21.15.67:6789/0 conn(0x7fd8081542f0 legacy=0x7fd8080ad3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:59.043 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd80cfdf700 1 -- 172.21.15.67:0/187493648 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8081546c0 con 0x7fd8080af7d0 2023-06-23T01:23:59.043 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd806ffd700 1 -- 172.21.15.67:0/187493648 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922171201 0 0) 0x7fd80806eee0 con 0x7fd8081542f0 2023-06-23T01:23:59.043 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd80cfdf700 1 --2- 172.21.15.67:0/187493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd8080aa870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd800009e60 tx=0x7fd800009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7041ea8c2e152bdc server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:59.044 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd806ffd700 1 -- 172.21.15.67:0/187493648 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80000e070 con 0x7fd8080af7d0 2023-06-23T01:23:59.044 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd806ffd700 1 -- 172.21.15.67:0/187493648 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd80000bb60 con 0x7fd8080af7d0 2023-06-23T01:23:59.044 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.035+0000 7fd806ffd700 1 -- 172.21.15.67:0/187493648 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800012430 con 0x7fd8080af7d0 2023-06-23T01:23:59.045 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.036+0000 7fd80ea42700 1 -- 172.21.15.67:0/187493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 msgr2=0x7fd8080aa870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:59.045 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.036+0000 7fd80ea42700 1 --2- 172.21.15.67:0/187493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd8080aa870 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd800009e60 tx=0x7fd800009780 comp rx=0 tx=0).stop 2023-06-23T01:23:59.048 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.036+0000 7fd80ea42700 1 -- 172.21.15.67:0/187493648 shutdown_connections 2023-06-23T01:23:59.048 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.036+0000 7fd80ea42700 1 --2- 172.21.15.67:0/187493648 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd8080aa870 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.048 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.036+0000 7fd80ea42700 1 --2- 172.21.15.67:0/187493648 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 0x7fd80815ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.048 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.036+0000 7fd80ea42700 1 -- 172.21.15.67:0/187493648 >> 172.21.15.67:0/187493648 conn(0x7fd8080cd060 msgr2=0x7fd8080cd460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:59.049 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.036+0000 7fd80ea42700 1 -- 172.21.15.67:0/187493648 shutdown_connections 2023-06-23T01:23:59.049 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.036+0000 7fd80ea42700 1 -- 172.21.15.67:0/187493648 wait complete. 2023-06-23T01:23:59.051 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.037+0000 7fd80ea42700 1 Processor -- start 2023-06-23T01:23:59.051 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.037+0000 7fd80ea42700 1 -- start start 2023-06-23T01:23:59.051 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd80ea42700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 0x7fd8080b1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:59.051 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd80ea42700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd808141a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:59.051 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd80ea42700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8081542f0 0x7fd808169eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:23:59.052 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd80ea42700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8080b23e0 con 0x7fd8080af7d0 2023-06-23T01:23:59.052 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd80ea42700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd80815cea0 con 0x7fd8081542f0 2023-06-23T01:23:59.052 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd80ea42700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd80815cba0 con 0x7fd8080adad0 2023-06-23T01:23:59.052 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd8077fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd808141a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:59.052 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd8077fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd808141a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56626/0 (socket says 172.21.15.67:56626) 2023-06-23T01:23:59.052 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd807fff700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 0x7fd8080b1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:59.052 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd8077fe700 1 -- 172.21.15.67:0/3759189299 learned_addr learned my addr 172.21.15.67:0/3759189299 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:23:59.053 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd80cfdf700 1 --2- 172.21.15.67:0/3759189299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8081542f0 0x7fd808169eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:23:59.053 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.038+0000 7fd8077fe700 1 -- 172.21.15.67:0/3759189299 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 msgr2=0x7fd8080b1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:59.053 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd8077fe700 1 --2- 172.21.15.67:0/3759189299 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 0x7fd8080b1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.053 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd8077fe700 1 -- 172.21.15.67:0/3759189299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8081542f0 msgr2=0x7fd808169eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:59.053 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd8077fe700 1 --2- 172.21.15.67:0/3759189299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8081542f0 0x7fd808169eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.053 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd8077fe700 1 -- 172.21.15.67:0/3759189299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd800009a20 con 0x7fd8080af7d0 2023-06-23T01:23:59.054 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd8077fe700 1 --2- 172.21.15.67:0/3759189299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd808141a60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000ecb0 tx=0x7fd7f00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:23:59.054 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd7fffff700 1 -- 172.21.15.67:0/3759189299 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f000c850 con 0x7fd8080af7d0 2023-06-23T01:23:59.054 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd7fffff700 1 -- 172.21.15.67:0/3759189299 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7f000c9b0 con 0x7fd8080af7d0 2023-06-23T01:23:59.054 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8080b2610 con 0x7fd8080af7d0 2023-06-23T01:23:59.054 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd7fffff700 1 -- 172.21.15.67:0/3759189299 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7f000cd50 con 0x7fd8080af7d0 2023-06-23T01:23:59.054 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8080b2bd0 con 0x7fd8080af7d0 2023-06-23T01:23:59.054 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd7fffff700 1 -- 172.21.15.67:0/3759189299 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd7f0005c80 con 0x7fd8080af7d0 2023-06-23T01:23:59.055 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.039+0000 7fd7fffff700 1 -- 172.21.15.67:0/3759189299 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 1879+0+0 (secure 0 0 0) 0x7fd7f00067f0 con 0x7fd8080af7d0 2023-06-23T01:23:59.055 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.040+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd80815d010 con 0x7fd8080af7d0 2023-06-23T01:23:59.055 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.042+0000 7fd7fffff700 1 -- 172.21.15.67:0/3759189299 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fd7f0021060 con 0x7fd8080af7d0 2023-06-23T01:23:59.081 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:59.079+0000 7f83299ed180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:23:59.102 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:59.099+0000 7fdc8d22c180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:23:59.115 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.112+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd80805f060 con 0x7fd8080af7d0 2023-06-23T01:23:59.115 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.113+0000 7fd7fffff700 1 -- 172.21.15.67:0/3759189299 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fd7f0017070 con 0x7fd8080af7d0 2023-06-23T01:23:59.115 INFO:teuthology.misc.health.smithi067.stdout: 2023-06-23T01:23:59.116 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":11,"fsid":"0068cd93-d7dd-4502-a85d-c8368ea7e9d2","created":"2023-06-23T01:23:54.450197+0000","modified":"2023-06-23T01:23:58.785762+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:23:57.782500+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.69999998807907104,"backfillfull_ratio":0.60000002384185791,"nearfull_ratio":0.60000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:23:59.117 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 msgr2=0x7fd808141a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:23:59.117 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 --2- 172.21.15.67:0/3759189299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd808141a60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd7f000ecb0 tx=0x7fd7f00091a0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.117 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 shutdown_connections 2023-06-23T01:23:59.117 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 --2- 172.21.15.67:0/3759189299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8081542f0 0x7fd808169eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.117 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 --2- 172.21.15.67:0/3759189299 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7fd8080adad0 0x7fd8080b1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.117 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 --2- 172.21.15.67:0/3759189299 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7fd8080af7d0 0x7fd808141a60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:23:59.117 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 >> 172.21.15.67:0/3759189299 conn(0x7fd8080cd060 msgr2=0x7fd80814fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:23:59.118 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 shutdown_connections 2023-06-23T01:23:59.118 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:23:59.114+0000 7fd80ea42700 1 -- 172.21.15.67:0/3759189299 wait complete. 2023-06-23T01:23:59.127 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-23T01:23:59.329 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:59.327+0000 7f83299ed180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:23:59.391 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:59.389+0000 7fdc8d22c180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:23:59.405 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:23:59.403+0000 7f83299ed180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:23:59.472 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:23:59.470+0000 7fdc8d22c180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:23:59.809 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-06-23T01:23:59.806+0000 7f29801bc700 -1 osd.2 0 waiting for initial osdmap 2023-06-23T01:23:59.809 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-06-23T01:23:59.807+0000 7efce435b700 -1 osd.3 0 waiting for initial osdmap 2023-06-23T01:23:59.810 INFO:tasks.ceph.osd.3.smithi067.stderr:2023-06-23T01:23:59.808+0000 7efcde154700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:23:59.812 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-06-23T01:23:59.810+0000 7f9c83d4f700 -1 osd.4 0 waiting for initial osdmap 2023-06-23T01:23:59.813 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-06-23T01:23:59.811+0000 7f9c621e9700 -1 osd.5 0 waiting for initial osdmap 2023-06-23T01:23:59.813 INFO:tasks.ceph.osd.4.smithi193.stderr:2023-06-23T01:23:59.811+0000 7f9c7d347700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:23:59.814 INFO:tasks.ceph.osd.2.smithi067.stderr:2023-06-23T01:23:59.812+0000 7f2979fb5700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:23:59.815 INFO:tasks.ceph.osd.5.smithi193.stderr:2023-06-23T01:23:59.813+0000 7f9c5bfe2700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:23:59.815 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-06-23T01:23:59.813+0000 7f8631342700 -1 osd.6 0 waiting for initial osdmap 2023-06-23T01:23:59.816 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-06-23T01:23:59.814+0000 7f47ef76d700 -1 osd.7 0 waiting for initial osdmap 2023-06-23T01:23:59.817 INFO:tasks.ceph.osd.6.smithi193.stderr:2023-06-23T01:23:59.814+0000 7f862b13b700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:23:59.817 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-06-23T01:23:59.815+0000 7f5da7aa4700 -1 osd.0 0 waiting for initial osdmap 2023-06-23T01:23:59.817 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-06-23T01:23:59.815+0000 7f6b22b34700 -1 osd.1 0 waiting for initial osdmap 2023-06-23T01:23:59.818 INFO:tasks.ceph.osd.7.smithi193.stderr:2023-06-23T01:23:59.815+0000 7f47e8d65700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:23:59.819 INFO:tasks.ceph.osd.1.smithi067.stderr:2023-06-23T01:23:59.817+0000 7f6b1c12c700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:23:59.819 INFO:tasks.ceph.osd.0.smithi067.stderr:2023-06-23T01:23:59.817+0000 7f5da109c700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:24:00.230 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.228+0000 7f83299ed180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:24:00.303 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.301+0000 7f83299ed180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:24:00.377 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.374+0000 7f83299ed180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:24:00.389 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:00.383+0000 7fdc8d22c180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:24:00.463 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:00.460+0000 7fdc8d22c180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:24:00.509 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.507+0000 7f83299ed180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:24:00.541 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:00.539+0000 7fdc8d22c180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:24:00.594 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.592+0000 7f83299ed180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:24:00.683 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:00.681+0000 7fdc8d22c180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:24:00.742 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.740+0000 7f83299ed180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:24:00.779 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:00.777+0000 7fdc8d22c180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:24:00.821 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.819+0000 7f83299ed180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:24:00.899 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.896+0000 7f83299ed180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:24:00.946 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:00.944+0000 7fdc8d22c180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:24:00.974 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:00.971+0000 7f83299ed180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:24:01.028 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:01.026+0000 7fdc8d22c180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:24:01.109 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:01.106+0000 7fdc8d22c180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:24:01.187 INFO:tasks.ceph.mgr.y.smithi067.stderr:2023-06-23T01:24:01.185+0000 7fdc8d22c180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:24:02.003 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:02.001+0000 7f82e2dee700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-06-23T01:24:02.004 INFO:tasks.ceph.mgr.x.smithi193.stderr:2023-06-23T01:24:02.001+0000 7f82e2dee700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-06-23T01:24:05.934 DEBUG:teuthology.orchestra.run.smithi067:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:24:06.056 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.053+0000 7f931150c700 1 Processor -- start 2023-06-23T01:24:06.056 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.054+0000 7f931150c700 1 -- start start 2023-06-23T01:24:06.058 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.054+0000 7f931150c700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f930c159d80 0x7f930c0cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.059 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.054+0000 7f931150c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f930c0cb240 0x7f930c068b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.059 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.054+0000 7f931150c700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c069090 0x7f930c0c6960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.059 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.054+0000 7f931150c700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c069510 con 0x7f930c0cb240 2023-06-23T01:24:06.059 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.054+0000 7f931150c700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c06c890 con 0x7f930c159d80 2023-06-23T01:24:06.059 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.054+0000 7f931150c700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0c64d0 con 0x7f930c069090 2023-06-23T01:24:06.059 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f930c0cb240 0x7f930c068b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.059 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f930c0cb240 0x7f930c068b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56642/0 (socket says 172.21.15.67:56642) 2023-06-23T01:24:06.060 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 -- 172.21.15.67:0/3436175479 learned_addr learned my addr 172.21.15.67:0/3436175479 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:06.060 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930affd700 1 --2- 172.21.15.67:0/3436175479 >> v2:172.21.15.193:3300/0 conn(0x7f930c159d80 0x7f930c0cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.060 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930b7fe700 1 --2- 172.21.15.67:0/3436175479 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c069090 0x7f930c0c6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.060 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 -- 172.21.15.67:0/3436175479 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c069090 msgr2=0x7f930c0c6960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:06.061 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 --2- 172.21.15.67:0/3436175479 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c069090 0x7f930c0c6960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.061 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 -- 172.21.15.67:0/3436175479 >> v2:172.21.15.193:3300/0 conn(0x7f930c159d80 msgr2=0x7f930c0cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:06.061 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 --2- 172.21.15.67:0/3436175479 >> v2:172.21.15.193:3300/0 conn(0x7f930c159d80 0x7f930c0cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.061 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 -- 172.21.15.67:0/3436175479 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0c5be0 con 0x7f930c0cb240 2023-06-23T01:24:06.061 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f930a7fc700 1 --2- 172.21.15.67:0/3436175479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c0cb240 0x7f930c068b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f930c06c580 tx=0x7f92f4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32283ba516276371 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:06.061 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f9309ffb700 1 -- 172.21.15.67:0/3436175479 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f400b820 con 0x7f930c0cb240 2023-06-23T01:24:06.061 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f9309ffb700 1 -- 172.21.15.67:0/3436175479 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f400b980 con 0x7f930c0cb240 2023-06-23T01:24:06.062 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.055+0000 7f9309ffb700 1 -- 172.21.15.67:0/3436175479 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f400bd20 con 0x7f930c0cb240 2023-06-23T01:24:06.062 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 -- 172.21.15.67:0/3436175479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c0cb240 msgr2=0x7f930c068b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.062 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 --2- 172.21.15.67:0/3436175479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c0cb240 0x7f930c068b50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f930c06c580 tx=0x7f92f4009250 comp rx=0 tx=0).stop 2023-06-23T01:24:06.062 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 -- 172.21.15.67:0/3436175479 shutdown_connections 2023-06-23T01:24:06.062 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 --2- 172.21.15.67:0/3436175479 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c069090 0x7f930c0c6960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.062 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 --2- 172.21.15.67:0/3436175479 >> v2:172.21.15.193:3300/0 conn(0x7f930c159d80 0x7f930c0cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.062 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 --2- 172.21.15.67:0/3436175479 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c0cb240 0x7f930c068b50 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.063 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 -- 172.21.15.67:0/3436175479 >> 172.21.15.67:0/3436175479 conn(0x7f930c067880 msgr2=0x7f930c067c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:06.063 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 -- 172.21.15.67:0/3436175479 shutdown_connections 2023-06-23T01:24:06.063 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.056+0000 7f931150c700 1 -- 172.21.15.67:0/3436175479 wait complete. 2023-06-23T01:24:06.065 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.057+0000 7f931150c700 1 Processor -- start 2023-06-23T01:24:06.065 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.057+0000 7f931150c700 1 -- start start 2023-06-23T01:24:06.065 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f931150c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c069090 0x7f930c155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.065 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f931150c700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c0cb240 0x7f930c169fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.066 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f931150c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f930c159d80 0x7f930c16a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.066 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f931150c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0b1c50 con 0x7f930c069090 2023-06-23T01:24:06.066 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f931150c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c16dc80 con 0x7f930c159d80 2023-06-23T01:24:06.066 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f931150c700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c16ae50 con 0x7f930c0cb240 2023-06-23T01:24:06.066 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c069090 0x7f930c155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.066 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c069090 0x7f930c155ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56650/0 (socket says 172.21.15.67:56650) 2023-06-23T01:24:06.066 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930affd700 1 -- 172.21.15.67:0/4045054007 learned_addr learned my addr 172.21.15.67:0/4045054007 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:06.067 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930b7fe700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f930c159d80 0x7f930c16a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.067 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930a7fc700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c0cb240 0x7f930c169fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.067 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930affd700 1 -- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c0cb240 msgr2=0x7f930c169fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.067 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930affd700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c0cb240 0x7f930c169fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.067 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930affd700 1 -- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f930c159d80 msgr2=0x7f930c16a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.067 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930affd700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f930c159d80 0x7f930c16a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.067 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.058+0000 7f930affd700 1 -- 172.21.15.67:0/4045054007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f40094f0 con 0x7f930c069090 2023-06-23T01:24:06.067 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.059+0000 7f930affd700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c069090 0x7f930c155ea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00d890 tx=0x7f92fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:06.068 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.059+0000 7f9302ffd700 1 -- 172.21.15.67:0/4045054007 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc0125f0 con 0x7f930c069090 2023-06-23T01:24:06.068 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.059+0000 7f9302ffd700 1 -- 172.21.15.67:0/4045054007 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92fc012750 con 0x7f930c069090 2023-06-23T01:24:06.068 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.059+0000 7f9302ffd700 1 -- 172.21.15.67:0/4045054007 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92fc012af0 con 0x7f930c069090 2023-06-23T01:24:06.068 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.059+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c1563e0 con 0x7f930c069090 2023-06-23T01:24:06.068 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.059+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c16d630 con 0x7f930c069090 2023-06-23T01:24:06.068 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.060+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c16df60 con 0x7f930c069090 2023-06-23T01:24:06.068 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.060+0000 7f9302ffd700 1 -- 172.21.15.67:0/4045054007 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f92fc01e460 con 0x7f930c069090 2023-06-23T01:24:06.069 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.061+0000 7f9302ffd700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f92ec066ec0 0x7f92ec069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.069 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.061+0000 7f9302ffd700 1 -- 172.21.15.67:0/4045054007 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f92fc0845f0 con 0x7f930c069090 2023-06-23T01:24:06.069 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.061+0000 7f930a7fc700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f92ec066ec0 0x7f92ec069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.069 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.061+0000 7f930a7fc700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f92ec066ec0 0x7f92ec069370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f930c06c580 tx=0x7f92f4017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:06.069 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.065+0000 7f9302ffd700 1 -- 172.21.15.67:0/4045054007 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f92fc051c70 con 0x7f930c069090 2023-06-23T01:24:06.088 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:24:06.090 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:24:06.264 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.262+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f930c05f060 con 0x7f930c069090 2023-06-23T01:24:06.265 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.263+0000 7f9302ffd700 1 -- 172.21.15.67:0/4045054007 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12175 (secure 0 0 0) 0x7f92fc055290 con 0x7f930c069090 2023-06-23T01:24:06.265 INFO:teuthology.misc.health.smithi067.stdout: 2023-06-23T01:24:06.265 INFO:teuthology.misc.health.smithi067.stdout:{"epoch":16,"fsid":"0068cd93-d7dd-4502-a85d-c8368ea7e9d2","created":"2023-06-23T01:23:54.450197+0000","modified":"2023-06-23T01:24:03.822529+0000","last_up_change":"2023-06-23T01:24:00.795797+0000","last_in_change":"2023-06-23T01:23:57.782500+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.69999998807907104,"backfillfull_ratio":0.60000002384185791,"nearfull_ratio":0.60000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:24:01.045792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":111550},{"type":"v1","addr":"172.21.15.67:6801","nonce":111550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":111550},{"type":"v1","addr":"172.21.15.67:6803","nonce":111550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":111550},{"type":"v1","addr":"172.21.15.67:6807","nonce":111550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":111550},{"type":"v1","addr":"172.21.15.67:6805","nonce":111550}]},"public_addr":"172.21.15.67:6801/111550","cluster_addr":"172.21.15.67:6803/111550","heartbeat_back_addr":"172.21.15.67:6807/111550","heartbeat_front_addr":"172.21.15.67:6805/111550","state":["exists","up"]},{"osd":1,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":111554},{"type":"v1","addr":"172.21.15.67:6812","nonce":111554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":111554},{"type":"v1","addr":"172.21.15.67:6816","nonce":111554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":111554},{"type":"v1","addr":"172.21.15.67:6823","nonce":111554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":111554},{"type":"v1","addr":"172.21.15.67:6820","nonce":111554}]},"public_addr":"172.21.15.67:6812/111554","cluster_addr":"172.21.15.67:6816/111554","heartbeat_back_addr":"172.21.15.67:6823/111554","heartbeat_front_addr":"172.21.15.67:6820/111554","state":["exists","up"]},{"osd":2,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":111558},{"type":"v1","addr":"172.21.15.67:6809","nonce":111558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":111558},{"type":"v1","addr":"172.21.15.67:6813","nonce":111558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":111558},{"type":"v1","addr":"172.21.15.67:6821","nonce":111558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":111558},{"type":"v1","addr":"172.21.15.67:6817","nonce":111558}]},"public_addr":"172.21.15.67:6809/111558","cluster_addr":"172.21.15.67:6813/111558","heartbeat_back_addr":"172.21.15.67:6821/111558","heartbeat_front_addr":"172.21.15.67:6817/111558","state":["exists","up"]},{"osd":3,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":111567},{"type":"v1","addr":"172.21.15.67:6825","nonce":111567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":111567},{"type":"v1","addr":"172.21.15.67:6827","nonce":111567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":111567},{"type":"v1","addr":"172.21.15.67:6831","nonce":111567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":111567},{"type":"v1","addr":"172.21.15.67:6829","nonce":111567}]},"public_addr":"172.21.15.67:6825/111567","cluster_addr":"172.21.15.67:6827/111567","heartbeat_back_addr":"172.21.15.67:6831/111567","heartbeat_front_addr":"172.21.15.67:6829/111567","state":["exists","up"]},{"osd":4,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":83673},{"type":"v1","addr":"172.21.15.193:6801","nonce":83673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":83673},{"type":"v1","addr":"172.21.15.193:6803","nonce":83673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":83673},{"type":"v1","addr":"172.21.15.193:6807","nonce":83673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":83673},{"type":"v1","addr":"172.21.15.193:6805","nonce":83673}]},"public_addr":"172.21.15.193:6801/83673","cluster_addr":"172.21.15.193:6803/83673","heartbeat_back_addr":"172.21.15.193:6807/83673","heartbeat_front_addr":"172.21.15.193:6805/83673","state":["exists","up"]},{"osd":5,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83674},{"type":"v1","addr":"172.21.15.193:6809","nonce":83674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83674},{"type":"v1","addr":"172.21.15.193:6811","nonce":83674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83674},{"type":"v1","addr":"172.21.15.193:6815","nonce":83674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83674},{"type":"v1","addr":"172.21.15.193:6813","nonce":83674}]},"public_addr":"172.21.15.193:6809/83674","cluster_addr":"172.21.15.193:6811/83674","heartbeat_back_addr":"172.21.15.193:6815/83674","heartbeat_front_addr":"172.21.15.193:6813/83674","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":83699},{"type":"v1","addr":"172.21.15.193:6825","nonce":83699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":83699},{"type":"v1","addr":"172.21.15.193:6827","nonce":83699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":83699},{"type":"v1","addr":"172.21.15.193:6831","nonce":83699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":83699},{"type":"v1","addr":"172.21.15.193:6829","nonce":83699}]},"public_addr":"172.21.15.193:6825/83699","cluster_addr":"172.21.15.193:6827/83699","heartbeat_back_addr":"172.21.15.193:6831/83699","heartbeat_front_addr":"172.21.15.193:6829/83699","state":["exists","up"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":83675},{"type":"v1","addr":"172.21.15.193:6817","nonce":83675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":83675},{"type":"v1","addr":"172.21.15.193:6819","nonce":83675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":83675},{"type":"v1","addr":"172.21.15.193:6823","nonce":83675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":83675},{"type":"v1","addr":"172.21.15.193:6821","nonce":83675}]},"public_addr":"172.21.15.193:6817/83675","cluster_addr":"172.21.15.193:6819/83675","heartbeat_back_addr":"172.21.15.193:6823/83675","heartbeat_front_addr":"172.21.15.193:6821/83675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.993918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.906223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.928802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.986013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.876314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.922270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.983980+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.902644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:24:06.267 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f92ec066ec0 msgr2=0x7f92ec069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.267 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f92ec066ec0 0x7f92ec069370 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f930c06c580 tx=0x7f92f4017000 comp rx=0 tx=0).stop 2023-06-23T01:24:06.268 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c069090 msgr2=0x7f930c155ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.268 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c069090 0x7f930c155ea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f92fc00d890 tx=0x7f92fc008040 comp rx=0 tx=0).stop 2023-06-23T01:24:06.268 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 shutdown_connections 2023-06-23T01:24:06.268 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f92ec066ec0 0x7f92ec069370 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.268 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f930c159d80 0x7f930c16a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.268 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f930c069090 0x7f930c155ea0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.268 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 --2- 172.21.15.67:0/4045054007 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f930c0cb240 0x7f930c169fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.269 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 >> 172.21.15.67:0/4045054007 conn(0x7f930c067880 msgr2=0x7f930c0ceb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:06.269 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 shutdown_connections 2023-06-23T01:24:06.269 INFO:teuthology.misc.health.smithi067.stderr:2023-06-23T01:24:06.265+0000 7f931150c700 1 -- 172.21.15.67:0/4045054007 wait complete. 2023-06-23T01:24:06.278 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-06-23T01:24:06.278 INFO:tasks.ceph:Creating RBD pool 2023-06-23T01:24:06.278 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd pool create rbd 8 2023-06-23T01:24:06.396 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.394+0000 7f523f1e6700 1 Processor -- start 2023-06-23T01:24:06.397 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.394+0000 7f523f1e6700 1 -- start start 2023-06-23T01:24:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523f1e6700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5238159ae0 0x7f52380c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523f1e6700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f52380cb8b0 0x7f523815dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523f1e6700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52380c5c30 con 0x7f5238159eb0 2023-06-23T01:24:06.398 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523f1e6700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523815e8f0 con 0x7f5238159ae0 2023-06-23T01:24:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523f1e6700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52380c5f10 con 0x7f52380cb8b0 2023-06-23T01:24:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523cf82700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5238159ae0 0x7f52380c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523cf82700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5238159ae0 0x7f52380c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56662/0 (socket says 172.21.15.67:56662) 2023-06-23T01:24:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523cf82700 1 -- 172.21.15.67:0/2521307311 learned_addr learned my addr 172.21.15.67:0/2521307311 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523d783700 1 --2- 172.21.15.67:0/2521307311 >> v2:172.21.15.193:3300/0 conn(0x7f52380cb8b0 0x7f523815dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523cf82700 1 -- 172.21.15.67:0/2521307311 >> v2:172.21.15.193:3300/0 conn(0x7f52380cb8b0 msgr2=0x7f523815dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523cf82700 1 --2- 172.21.15.67:0/2521307311 >> v2:172.21.15.193:3300/0 conn(0x7f52380cb8b0 0x7f523815dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.399 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523cf82700 1 -- 172.21.15.67:0/2521307311 >> v1:172.21.15.193:6789/0 conn(0x7f5238159eb0 legacy=0x7f52380cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.395+0000 7f523cf82700 1 -- 172.21.15.67:0/2521307311 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52380c61f0 con 0x7f5238159ae0 2023-06-23T01:24:06.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523cf82700 1 --2- 172.21.15.67:0/2521307311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159ae0 0x7f52380c69f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f52380ccc90 tx=0x7f5230009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da88db8b33eb497a server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:06.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f52377fe700 1 -- 172.21.15.67:0/2521307311 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523000e070 con 0x7f5238159ae0 2023-06-23T01:24:06.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f52377fe700 1 -- 172.21.15.67:0/2521307311 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523000ba10 con 0x7f5238159ae0 2023-06-23T01:24:06.400 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f52377fe700 1 -- 172.21.15.67:0/2521307311 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52300092e0 con 0x7f5238159ae0 2023-06-23T01:24:06.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523f1e6700 1 -- 172.21.15.67:0/2521307311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159ae0 msgr2=0x7f52380c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523f1e6700 1 --2- 172.21.15.67:0/2521307311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159ae0 0x7f52380c69f0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f52380ccc90 tx=0x7f5230009840 comp rx=0 tx=0).stop 2023-06-23T01:24:06.402 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523f1e6700 1 -- 172.21.15.67:0/2521307311 shutdown_connections 2023-06-23T01:24:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523f1e6700 1 --2- 172.21.15.67:0/2521307311 >> v2:172.21.15.193:3300/0 conn(0x7f52380cb8b0 0x7f523815dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523f1e6700 1 --2- 172.21.15.67:0/2521307311 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159ae0 0x7f52380c69f0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523f1e6700 1 -- 172.21.15.67:0/2521307311 >> 172.21.15.67:0/2521307311 conn(0x7f5238063020 msgr2=0x7f5238063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523f1e6700 1 -- 172.21.15.67:0/2521307311 shutdown_connections 2023-06-23T01:24:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.396+0000 7f523f1e6700 1 -- 172.21.15.67:0/2521307311 wait complete. 2023-06-23T01:24:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.397+0000 7f523f1e6700 1 Processor -- start 2023-06-23T01:24:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.398+0000 7f523f1e6700 1 -- start start 2023-06-23T01:24:06.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.398+0000 7f523f1e6700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52380cb8b0 0x7f5238156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.398+0000 7f523f1e6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5238159ae0 0x7f52380d1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.398+0000 7f523f1e6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159eb0 0x7f5238158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.398+0000 7f523f1e6700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52380cf4a0 con 0x7f5238159eb0 2023-06-23T01:24:06.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.398+0000 7f523f1e6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523815af50 con 0x7f5238159ae0 2023-06-23T01:24:06.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.398+0000 7f523f1e6700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523815b510 con 0x7f52380cb8b0 2023-06-23T01:24:06.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159eb0 0x7f5238158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.404 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159eb0 0x7f5238158440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56668/0 (socket says 172.21.15.67:56668) 2023-06-23T01:24:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 -- 172.21.15.67:0/41240961 learned_addr learned my addr 172.21.15.67:0/41240961 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f5237fff700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5238159ae0 0x7f52380d1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523cf82700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52380cb8b0 0x7f5238156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 -- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52380cb8b0 msgr2=0x7f5238156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52380cb8b0 0x7f5238156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 -- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5238159ae0 msgr2=0x7f52380d1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.405 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5238159ae0 0x7f52380d1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 -- 172.21.15.67:0/41240961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5230009ae0 con 0x7f5238159eb0 2023-06-23T01:24:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523cf82700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52380cb8b0 0x7f5238156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:24:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523d783700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159eb0 0x7f5238158440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f522800d8c0 tx=0x7f5228008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f5234ff9700 1 -- 172.21.15.67:0/41240961 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52280125f0 con 0x7f5238159eb0 2023-06-23T01:24:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f5234ff9700 1 -- 172.21.15.67:0/41240961 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5228012770 con 0x7f5238159eb0 2023-06-23T01:24:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52380cf6f0 con 0x7f5238159eb0 2023-06-23T01:24:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f5234ff9700 1 -- 172.21.15.67:0/41240961 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5228012b30 con 0x7f5238159eb0 2023-06-23T01:24:06.406 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.399+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523815b1a0 con 0x7f5238159eb0 2023-06-23T01:24:06.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.400+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52380d1050 con 0x7f5238159eb0 2023-06-23T01:24:06.407 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.400+0000 7f5234ff9700 1 -- 172.21.15.67:0/41240961 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f522801e460 con 0x7f5238159eb0 2023-06-23T01:24:06.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.401+0000 7f5234ff9700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f5224066e90 0x7f5224069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.401+0000 7f5234ff9700 1 -- 172.21.15.67:0/41240961 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f5228084710 con 0x7f5238159eb0 2023-06-23T01:24:06.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.401+0000 7f523cf82700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f5224066e90 0x7f5224069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.401+0000 7f523cf82700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f5224066e90 0x7f5224069340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5230009880 tx=0x7f5230017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:06.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.405+0000 7f5234ff9700 1 -- 172.21.15.67:0/41240961 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f5228051d90 con 0x7f5238159eb0 2023-06-23T01:24:06.602 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.600+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f523805f060 con 0x7f5238159eb0 2023-06-23T01:24:06.855 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.852+0000 7f5234ff9700 1 -- 172.21.15.67:0/41240961 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f52280553b0 con 0x7f5238159eb0 2023-06-23T01:24:06.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f5224066e90 msgr2=0x7f5224069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f5224066e90 0x7f5224069340 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5230009880 tx=0x7f5230017000 comp rx=0 tx=0).stop 2023-06-23T01:24:06.858 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159eb0 msgr2=0x7f5238158440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159eb0 0x7f5238158440 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f522800d8c0 tx=0x7f5228008040 comp rx=0 tx=0).stop 2023-06-23T01:24:06.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 shutdown_connections 2023-06-23T01:24:06.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f5224066e90 0x7f5224069340 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5238159eb0 0x7f5238158440 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f52380cb8b0 0x7f5238156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 --2- 172.21.15.67:0/41240961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5238159ae0 0x7f52380d1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.859 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.855+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 >> 172.21.15.67:0/41240961 conn(0x7f5238063020 msgr2=0x7f523806d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:06.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.856+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 shutdown_connections 2023-06-23T01:24:06.860 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.856+0000 7f523f1e6700 1 -- 172.21.15.67:0/41240961 wait complete. 2023-06-23T01:24:06.860 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'rbd' created 2023-06-23T01:24:06.870 DEBUG:teuthology.orchestra.run.smithi067:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-06-23T01:24:06.990 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.988+0000 7f2a78e95700 1 Processor -- start 2023-06-23T01:24:06.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.988+0000 7f2a78e95700 1 -- start start 2023-06-23T01:24:06.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a78e95700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a7406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:06.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a78e95700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7415e820 con 0x7f2a740d0b70 2023-06-23T01:24:06.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a78e95700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7415d970 con 0x7f2a7406dd50 2023-06-23T01:24:06.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a78e95700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a740b55e0 con 0x7f2a74159c20 2023-06-23T01:24:06.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a72ffd700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f2a740d0b70 0x7f2a74159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:37260/0 (socket says 172.21.15.67:37260) 2023-06-23T01:24:06.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a72ffd700 1 -- 172.21.15.67:0/838110747 learned_addr learned my addr 172.21.15.67:0/838110747 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:06.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a737fe700 1 --2- 172.21.15.67:0/838110747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a7406e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:06.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a727fc700 1 -- 172.21.15.67:0/838110747 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821549772 0 0) 0x7f2a7415e820 con 0x7f2a740d0b70 2023-06-23T01:24:06.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a727fc700 1 -- 172.21.15.67:0/838110747 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a5c003570 con 0x7f2a740d0b70 2023-06-23T01:24:06.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a727fc700 1 -- 172.21.15.67:0/838110747 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277254009 0 0) 0x7f2a7415d970 con 0x7f2a7406dd50 2023-06-23T01:24:06.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a727fc700 1 -- 172.21.15.67:0/838110747 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a7415e820 con 0x7f2a7406dd50 2023-06-23T01:24:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.989+0000 7f2a737fe700 1 -- 172.21.15.67:0/838110747 >> v1:172.21.15.193:6789/0 conn(0x7f2a7406dd50 legacy=0x7f2a7415a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a737fe700 1 -- 172.21.15.67:0/838110747 >> v1:172.21.15.67:6789/0 conn(0x7f2a740d0b70 legacy=0x7f2a74159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a737fe700 1 -- 172.21.15.67:0/838110747 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7415dc10 con 0x7f2a74159c20 2023-06-23T01:24:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a737fe700 1 --2- 172.21.15.67:0/838110747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a7406e9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009910 tx=0x7f2a640091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c90ea2929b7adec server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a727fc700 1 -- 172.21.15.67:0/838110747 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64011630 con 0x7f2a74159c20 2023-06-23T01:24:06.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a727fc700 1 -- 172.21.15.67:0/838110747 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64011790 con 0x7f2a74159c20 2023-06-23T01:24:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a727fc700 1 -- 172.21.15.67:0/838110747 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64011b30 con 0x7f2a74159c20 2023-06-23T01:24:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a78e95700 1 -- 172.21.15.67:0/838110747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 msgr2=0x7f2a7406e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a78e95700 1 --2- 172.21.15.67:0/838110747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a7406e9e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2a64009910 tx=0x7f2a640091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a78e95700 1 -- 172.21.15.67:0/838110747 shutdown_connections 2023-06-23T01:24:06.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a78e95700 1 --2- 172.21.15.67:0/838110747 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a7406e9e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:06.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a78e95700 1 -- 172.21.15.67:0/838110747 >> 172.21.15.67:0/838110747 conn(0x7f2a74063020 msgr2=0x7f2a74063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:06.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a78e95700 1 -- 172.21.15.67:0/838110747 shutdown_connections 2023-06-23T01:24:06.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.990+0000 7f2a78e95700 1 -- 172.21.15.67:0/838110747 wait complete. 2023-06-23T01:24:06.999 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a78e95700 1 Processor -- start 2023-06-23T01:24:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a78e95700 1 -- start start 2023-06-23T01:24:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a78e95700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7406dd50 0x7f2a74155fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a78e95700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a740d0b70 0x7f2a740b6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a78e95700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a74142200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a78e95700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a74154dd0 con 0x7f2a740d0b70 2023-06-23T01:24:07.000 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a78e95700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a74156620 con 0x7f2a7406dd50 2023-06-23T01:24:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a78e95700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a74156900 con 0x7f2a74159c20 2023-06-23T01:24:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a72ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a740d0b70 0x7f2a740b6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a72ffd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a740d0b70 0x7f2a740b6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:56684/0 (socket says 172.21.15.67:56684) 2023-06-23T01:24:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.992+0000 7f2a72ffd700 1 -- 172.21.15.67:0/3148214800 learned_addr learned my addr 172.21.15.67:0/3148214800 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a737fe700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7406dd50 0x7f2a74155fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a73fff700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a74142200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:07.001 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a72ffd700 1 -- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 msgr2=0x7f2a74142200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:07.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a72ffd700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a74142200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:07.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a72ffd700 1 -- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7406dd50 msgr2=0x7f2a74155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:07.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a72ffd700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7406dd50 0x7f2a74155fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:07.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a72ffd700 1 -- 172.21.15.67:0/3148214800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a64009420 con 0x7f2a740d0b70 2023-06-23T01:24:07.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a72ffd700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a740d0b70 0x7f2a740b6110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c790 tx=0x7f2a68007e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:07.002 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a5b7fe700 1 -- 172.21.15.67:0/3148214800 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68005b30 con 0x7f2a740d0b70 2023-06-23T01:24:07.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a5b7fe700 1 -- 172.21.15.67:0/3148214800 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a68005cd0 con 0x7f2a740d0b70 2023-06-23T01:24:07.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a5b7fe700 1 -- 172.21.15.67:0/3148214800 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a68007320 con 0x7f2a740d0b70 2023-06-23T01:24:07.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a74156be0 con 0x7f2a740d0b70 2023-06-23T01:24:07.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.993+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a741571a0 con 0x7f2a740d0b70 2023-06-23T01:24:07.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.994+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a74157360 con 0x7f2a740d0b70 2023-06-23T01:24:07.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.995+0000 7f2a5b7fe700 1 -- 172.21.15.67:0/3148214800 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f2a68012400 con 0x7f2a740d0b70 2023-06-23T01:24:07.003 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.995+0000 7f2a5b7fe700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2a60066e90 0x7f2a60069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:07.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.995+0000 7f2a5b7fe700 1 -- 172.21.15.67:0/3148214800 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5195+0+0 (secure 0 0 0) 0x7f2a68083600 con 0x7f2a740d0b70 2023-06-23T01:24:07.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.995+0000 7f2a737fe700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2a60066e90 0x7f2a60069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:07.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:06.996+0000 7f2a737fe700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2a60066e90 0x7f2a60069340 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2a640091e0 tx=0x7f2a64017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:07.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.000+0000 7f2a5b7fe700 1 -- 172.21.15.67:0/3148214800 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f2a6804b590 con 0x7f2a740d0b70 2023-06-23T01:24:07.197 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.195+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2a7405f060 con 0x7f2a740d0b70 2023-06-23T01:24:07.864 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.862+0000 7f2a5b7fe700 1 -- 172.21.15.67:0/3148214800 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f2a680545d0 con 0x7f2a740d0b70 2023-06-23T01:24:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2a60066e90 msgr2=0x7f2a60069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2a60066e90 0x7f2a60069340 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f2a640091e0 tx=0x7f2a64017000 comp rx=0 tx=0).stop 2023-06-23T01:24:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a740d0b70 msgr2=0x7f2a740b6110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a740d0b70 0x7f2a740b6110 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2a6800c790 tx=0x7f2a68007e80 comp rx=0 tx=0).stop 2023-06-23T01:24:07.867 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 shutdown_connections 2023-06-23T01:24:07.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2a60066e90 0x7f2a60069340 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:07.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2a74159c20 0x7f2a74142200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:07.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a7406dd50 0x7f2a74155fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:07.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 --2- 172.21.15.67:0/3148214800 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f2a740d0b70 0x7f2a740b6110 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:07.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 >> 172.21.15.67:0/3148214800 conn(0x7f2a74063020 msgr2=0x7f2a741538e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:07.868 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 shutdown_connections 2023-06-23T01:24:07.869 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:07.864+0000 7f2a78e95700 1 -- 172.21.15.67:0/3148214800 wait complete. 2023-06-23T01:24:07.869 INFO:teuthology.orchestra.run.smithi067.stderr:enabled application 'rbd' on pool 'rbd' 2023-06-23T01:24:07.880 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-06-23T01:24:07.880 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-23T01:24:07.881 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-23T01:24:07.883 INFO:tasks.ceph.mds.a:Started 2023-06-23T01:24:07.883 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-23T01:24:07.883 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-23T01:24:07.885 INFO:tasks.ceph.mds.c:Started 2023-06-23T01:24:07.885 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-23T01:24:07.886 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-23T01:24:07.888 INFO:tasks.ceph.mds.b:Started 2023-06-23T01:24:07.888 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-23T01:24:07.889 DEBUG:teuthology.orchestra.run.smithi193:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-23T01:24:07.891 INFO:tasks.ceph.mds.d:Started 2023-06-23T01:24:07.891 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-06-23T01:24:07.891 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:24:07.963 INFO:tasks.ceph.mds.a.smithi067.stdout:starting mds.a at 2023-06-23T01:24:07.963 INFO:tasks.ceph.mds.c.smithi067.stdout:starting mds.c at 2023-06-23T01:24:07.969 INFO:tasks.ceph.mds.b.smithi193.stdout:starting mds.b at 2023-06-23T01:24:07.970 INFO:tasks.ceph.mds.d.smithi193.stdout:starting mds.d at 2023-06-23T01:24:07.970 INFO:tasks.ceph.mds.b.smithi193.stderr:2023-06-23T01:24:07.968+0000 7f6844001700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2023-06-23T01:24:07.971 INFO:tasks.ceph.mds.d.smithi193.stderr:2023-06-23T01:24:07.969+0000 7f0cfb1cf700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6836/0: (98) Address already in use 2023-06-23T01:24:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.017+0000 7f21481db700 1 Processor -- start 2023-06-23T01:24:08.020 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.017+0000 7f21481db700 1 -- start start 2023-06-23T01:24:08.021 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.018+0000 7f21481db700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159eb0 0x7f21400cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.018+0000 7f21481db700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f21400cb6e0 0x7f214015dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.018+0000 7f21481db700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21400cbb00 con 0x7f2140159ae0 2023-06-23T01:24:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.018+0000 7f21481db700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f214015e8f0 con 0x7f21400cb6e0 2023-06-23T01:24:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.018+0000 7f21481db700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21400c5c30 con 0x7f2140159eb0 2023-06-23T01:24:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.018+0000 7f2145776700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159eb0 0x7f21400cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.018+0000 7f2145f77700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f2140159ae0 0x7f21400c69f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.67:38740/0 (socket says 172.21.15.67:38740) 2023-06-23T01:24:08.022 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.018+0000 7f2146778700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f21400cb6e0 0x7f214015dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2145f77700 1 -- 172.21.15.67:0/2313016045 learned_addr learned my addr 172.21.15.67:0/2313016045 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2144f75700 1 -- 172.21.15.67:0/2313016045 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4201393682 0 0) 0x7f21400cbb00 con 0x7f2140159ae0 2023-06-23T01:24:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2144f75700 1 -- 172.21.15.67:0/2313016045 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f212c003570 con 0x7f2140159ae0 2023-06-23T01:24:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2146778700 1 -- 172.21.15.67:0/2313016045 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159eb0 msgr2=0x7f21400cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2146778700 1 --2- 172.21.15.67:0/2313016045 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159eb0 0x7f21400cb1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2146778700 1 -- 172.21.15.67:0/2313016045 >> v1:172.21.15.193:6789/0 conn(0x7f2140159ae0 legacy=0x7f21400c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.023 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2146778700 1 -- 172.21.15.67:0/2313016045 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21400c5f10 con 0x7f21400cb6e0 2023-06-23T01:24:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2146778700 1 --2- 172.21.15.67:0/2313016045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21400cb6e0 0x7f214015dba0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2138009e60 tx=0x7f2138009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec642dd6986dc81e server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2144f75700 1 -- 172.21.15.67:0/2313016045 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213800e070 con 0x7f21400cb6e0 2023-06-23T01:24:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2144f75700 1 -- 172.21.15.67:0/2313016045 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213800bab0 con 0x7f21400cb6e0 2023-06-23T01:24:08.024 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.019+0000 7f2144f75700 1 -- 172.21.15.67:0/2313016045 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213800be70 con 0x7f21400cb6e0 2023-06-23T01:24:08.026 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.020+0000 7f21481db700 1 -- 172.21.15.67:0/2313016045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21400cb6e0 msgr2=0x7f214015dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.020+0000 7f21481db700 1 --2- 172.21.15.67:0/2313016045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21400cb6e0 0x7f214015dba0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2138009e60 tx=0x7f2138009780 comp rx=0 tx=0).stop 2023-06-23T01:24:08.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.020+0000 7f21481db700 1 -- 172.21.15.67:0/2313016045 shutdown_connections 2023-06-23T01:24:08.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.020+0000 7f21481db700 1 --2- 172.21.15.67:0/2313016045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f21400cb6e0 0x7f214015dba0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.020+0000 7f21481db700 1 --2- 172.21.15.67:0/2313016045 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159eb0 0x7f21400cb1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.020+0000 7f21481db700 1 -- 172.21.15.67:0/2313016045 >> 172.21.15.67:0/2313016045 conn(0x7f2140063020 msgr2=0x7f2140063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:08.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.020+0000 7f21481db700 1 -- 172.21.15.67:0/2313016045 shutdown_connections 2023-06-23T01:24:08.027 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.020+0000 7f21481db700 1 -- 172.21.15.67:0/2313016045 wait complete. 2023-06-23T01:24:08.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.021+0000 7f21481db700 1 Processor -- start 2023-06-23T01:24:08.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.021+0000 7f21481db700 1 -- start start 2023-06-23T01:24:08.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f21481db700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21400cb6e0 0x7f2140154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f21481db700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159ae0 0x7f21400cf760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f21481db700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2140159eb0 0x7f21400b4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f21481db700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214015b430 con 0x7f21400cb6e0 2023-06-23T01:24:08.028 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f21481db700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214015a820 con 0x7f2140159eb0 2023-06-23T01:24:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f21481db700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f214015ab00 con 0x7f2140159ae0 2023-06-23T01:24:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145f77700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21400cb6e0 0x7f2140154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159ae0 0x7f21400cf760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159ae0 0x7f21400cf760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.67:57484/0 (socket says 172.21.15.67:57484) 2023-06-23T01:24:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 -- 172.21.15.67:0/1316080921 learned_addr learned my addr 172.21.15.67:0/1316080921 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2146778700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2140159eb0 0x7f21400b4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 -- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2140159eb0 msgr2=0x7f21400b4ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.029 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2140159eb0 0x7f21400b4ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 -- 172.21.15.67:0/1316080921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21400cb6e0 msgr2=0x7f2140154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21400cb6e0 0x7f2140154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 -- 172.21.15.67:0/1316080921 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2138009a20 con 0x7f2140159ae0 2023-06-23T01:24:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145f77700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21400cb6e0 0x7f2140154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:24:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.022+0000 7f2145776700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159ae0 0x7f21400cf760 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f213c00ec80 tx=0x7f213c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.023+0000 7f2131ffb700 1 -- 172.21.15.67:0/1316080921 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c00c850 con 0x7f2140159ae0 2023-06-23T01:24:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.023+0000 7f2131ffb700 1 -- 172.21.15.67:0/1316080921 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f213c00c9b0 con 0x7f2140159ae0 2023-06-23T01:24:08.030 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.023+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214015ade0 con 0x7f2140159ae0 2023-06-23T01:24:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.023+0000 7f2131ffb700 1 -- 172.21.15.67:0/1316080921 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f213c00cd50 con 0x7f2140159ae0 2023-06-23T01:24:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.023+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2140154c70 con 0x7f2140159ae0 2023-06-23T01:24:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.024+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214005f060 con 0x7f2140159ae0 2023-06-23T01:24:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.024+0000 7f2131ffb700 1 -- 172.21.15.67:0/1316080921 <== mon.2 v2:172.21.15.193:3301/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f213c013680 con 0x7f2140159ae0 2023-06-23T01:24:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.024+0000 7f2131ffb700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2124066e90 0x7f2124069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.024+0000 7f2131ffb700 1 -- 172.21.15.67:0/1316080921 <== mon.2 v2:172.21.15.193:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f213c084500 con 0x7f2140159ae0 2023-06-23T01:24:08.031 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.024+0000 7f2145f77700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2124066e90 0x7f2124069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.025+0000 7f2145f77700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2124066e90 0x7f2124069340 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2134007fd0 tx=0x7f2134009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.032 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.029+0000 7f2131ffb700 1 -- 172.21.15.67:0/1316080921 <== mon.2 v2:172.21.15.193:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f213c051a20 con 0x7f2140159ae0 2023-06-23T01:24:08.225 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.223+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f214015b570 con 0x7f2140159ae0 2023-06-23T01:24:08.226 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.224+0000 7f2131ffb700 1 -- 172.21.15.67:0/1316080921 <== mon.2 v2:172.21.15.193:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+13708 (secure 0 0 0) 0x7f213c055040 con 0x7f2140159ae0 2023-06-23T01:24:08.226 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:24:08.226 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":18,"fsid":"0068cd93-d7dd-4502-a85d-c8368ea7e9d2","created":"2023-06-23T01:23:54.450197+0000","modified":"2023-06-23T01:24:07.849792+0000","last_up_change":"2023-06-23T01:24:00.795797+0000","last_in_change":"2023-06-23T01:23:57.782500+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.69999998807907104,"backfillfull_ratio":0.60000002384185791,"nearfull_ratio":0.60000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:24:01.045792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:24:06.629871+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":111550},{"type":"v1","addr":"172.21.15.67:6801","nonce":111550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":111550},{"type":"v1","addr":"172.21.15.67:6803","nonce":111550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":111550},{"type":"v1","addr":"172.21.15.67:6807","nonce":111550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":111550},{"type":"v1","addr":"172.21.15.67:6805","nonce":111550}]},"public_addr":"172.21.15.67:6801/111550","cluster_addr":"172.21.15.67:6803/111550","heartbeat_back_addr":"172.21.15.67:6807/111550","heartbeat_front_addr":"172.21.15.67:6805/111550","state":["exists","up"]},{"osd":1,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":111554},{"type":"v1","addr":"172.21.15.67:6812","nonce":111554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":111554},{"type":"v1","addr":"172.21.15.67:6816","nonce":111554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":111554},{"type":"v1","addr":"172.21.15.67:6823","nonce":111554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":111554},{"type":"v1","addr":"172.21.15.67:6820","nonce":111554}]},"public_addr":"172.21.15.67:6812/111554","cluster_addr":"172.21.15.67:6816/111554","heartbeat_back_addr":"172.21.15.67:6823/111554","heartbeat_front_addr":"172.21.15.67:6820/111554","state":["exists","up"]},{"osd":2,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":111558},{"type":"v1","addr":"172.21.15.67:6809","nonce":111558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":111558},{"type":"v1","addr":"172.21.15.67:6813","nonce":111558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":111558},{"type":"v1","addr":"172.21.15.67:6821","nonce":111558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":111558},{"type":"v1","addr":"172.21.15.67:6817","nonce":111558}]},"public_addr":"172.21.15.67:6809/111558","cluster_addr":"172.21.15.67:6813/111558","heartbeat_back_addr":"172.21.15.67:6821/111558","heartbeat_front_addr":"172.21.15.67:6817/111558","state":["exists","up"]},{"osd":3,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":111567},{"type":"v1","addr":"172.21.15.67:6825","nonce":111567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":111567},{"type":"v1","addr":"172.21.15.67:6827","nonce":111567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":111567},{"type":"v1","addr":"172.21.15.67:6831","nonce":111567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":111567},{"type":"v1","addr":"172.21.15.67:6829","nonce":111567}]},"public_addr":"172.21.15.67:6825/111567","cluster_addr":"172.21.15.67:6827/111567","heartbeat_back_addr":"172.21.15.67:6831/111567","heartbeat_front_addr":"172.21.15.67:6829/111567","state":["exists","up"]},{"osd":4,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":83673},{"type":"v1","addr":"172.21.15.193:6801","nonce":83673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":83673},{"type":"v1","addr":"172.21.15.193:6803","nonce":83673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":83673},{"type":"v1","addr":"172.21.15.193:6807","nonce":83673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":83673},{"type":"v1","addr":"172.21.15.193:6805","nonce":83673}]},"public_addr":"172.21.15.193:6801/83673","cluster_addr":"172.21.15.193:6803/83673","heartbeat_back_addr":"172.21.15.193:6807/83673","heartbeat_front_addr":"172.21.15.193:6805/83673","state":["exists","up"]},{"osd":5,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83674},{"type":"v1","addr":"172.21.15.193:6809","nonce":83674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83674},{"type":"v1","addr":"172.21.15.193:6811","nonce":83674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83674},{"type":"v1","addr":"172.21.15.193:6815","nonce":83674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83674},{"type":"v1","addr":"172.21.15.193:6813","nonce":83674}]},"public_addr":"172.21.15.193:6809/83674","cluster_addr":"172.21.15.193:6811/83674","heartbeat_back_addr":"172.21.15.193:6815/83674","heartbeat_front_addr":"172.21.15.193:6813/83674","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":83699},{"type":"v1","addr":"172.21.15.193:6825","nonce":83699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":83699},{"type":"v1","addr":"172.21.15.193:6827","nonce":83699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":83699},{"type":"v1","addr":"172.21.15.193:6831","nonce":83699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":83699},{"type":"v1","addr":"172.21.15.193:6829","nonce":83699}]},"public_addr":"172.21.15.193:6825/83699","cluster_addr":"172.21.15.193:6827/83699","heartbeat_back_addr":"172.21.15.193:6831/83699","heartbeat_front_addr":"172.21.15.193:6829/83699","state":["exists","up"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":83675},{"type":"v1","addr":"172.21.15.193:6817","nonce":83675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":83675},{"type":"v1","addr":"172.21.15.193:6819","nonce":83675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":83675},{"type":"v1","addr":"172.21.15.193:6823","nonce":83675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":83675},{"type":"v1","addr":"172.21.15.193:6821","nonce":83675}]},"public_addr":"172.21.15.193:6817/83675","cluster_addr":"172.21.15.193:6819/83675","heartbeat_back_addr":"172.21.15.193:6823/83675","heartbeat_front_addr":"172.21.15.193:6821/83675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.993918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.906223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.928802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.986013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.876314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.922270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.983980+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.902644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:24:08.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2124066e90 msgr2=0x7f2124069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2124066e90 0x7f2124069340 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2134007fd0 tx=0x7f2134009040 comp rx=0 tx=0).stop 2023-06-23T01:24:08.228 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159ae0 msgr2=0x7f21400cf760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159ae0 0x7f21400cf760 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f213c00ec80 tx=0x7f213c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 shutdown_connections 2023-06-23T01:24:08.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f2124066e90 0x7f2124069340 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2140159eb0 0x7f21400b4ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.229 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f21400cb6e0 0x7f2140154730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 --2- 172.21.15.67:0/1316080921 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f2140159ae0 0x7f21400cf760 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 >> 172.21.15.67:0/1316080921 conn(0x7f2140063020 msgr2=0x7f21400cd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 shutdown_connections 2023-06-23T01:24:08.230 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.226+0000 7f21481db700 1 -- 172.21.15.67:0/1316080921 wait complete. 2023-06-23T01:24:08.240 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:24:01.045792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-23T01:24:06.629871+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-06-23T01:24:08.241 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:24:08.362 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.359+0000 7efe6229f700 1 Processor -- start 2023-06-23T01:24:08.362 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.360+0000 7efe6229f700 1 -- start start 2023-06-23T01:24:08.364 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.360+0000 7efe6229f700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7efe5c1599f0 0x7efe5c06d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.364 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.360+0000 7efe6229f700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe5c063730 0x7efe5c0c9af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.364 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.360+0000 7efe6229f700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5c0c9660 con 0x7efe5c06dee0 2023-06-23T01:24:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.360+0000 7efe6229f700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5c069d70 con 0x7efe5c063730 2023-06-23T01:24:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.360+0000 7efe6229f700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe5c06e440 con 0x7efe5c1599f0 2023-06-23T01:24:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe6083c700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7efe5c063730 0x7efe5c0c9af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe5b7fe700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7efe5c06dee0 0x7efe5c063020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60012/0 (socket says 172.21.15.67:60012) 2023-06-23T01:24:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe5b7fe700 1 -- 172.21.15.67:0/2477849175 learned_addr learned my addr 172.21.15.67:0/2477849175 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:08.365 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe5bfff700 1 --2- 172.21.15.67:0/2477849175 >> v2:172.21.15.193:3300/0 conn(0x7efe5c1599f0 0x7efe5c06d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe6083c700 1 -- 172.21.15.67:0/2477849175 >> v2:172.21.15.193:3300/0 conn(0x7efe5c1599f0 msgr2=0x7efe5c06d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe6083c700 1 --2- 172.21.15.67:0/2477849175 >> v2:172.21.15.193:3300/0 conn(0x7efe5c1599f0 0x7efe5c06d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe6083c700 1 -- 172.21.15.67:0/2477849175 >> v1:172.21.15.67:6789/0 conn(0x7efe5c06dee0 legacy=0x7efe5c063020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe6083c700 1 -- 172.21.15.67:0/2477849175 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5c06e720 con 0x7efe5c063730 2023-06-23T01:24:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe5affd700 1 -- 172.21.15.67:0/2477849175 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1168004952 0 0) 0x7efe5c0c9660 con 0x7efe5c06dee0 2023-06-23T01:24:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe5bfff700 1 --2- 172.21.15.67:0/2477849175 >> v2:172.21.15.193:3300/0 conn(0x7efe5c1599f0 0x7efe5c06d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:24:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe6083c700 1 --2- 172.21.15.67:0/2477849175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c063730 0x7efe5c0c9af0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efe5c06d3b0 tx=0x7efe54009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c602a33116489ac3 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.366 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe5affd700 1 -- 172.21.15.67:0/2477849175 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe5400e070 con 0x7efe5c063730 2023-06-23T01:24:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe5affd700 1 -- 172.21.15.67:0/2477849175 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe5400bc30 con 0x7efe5c063730 2023-06-23T01:24:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.361+0000 7efe5affd700 1 -- 172.21.15.67:0/2477849175 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe54012430 con 0x7efe5c063730 2023-06-23T01:24:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.362+0000 7efe6229f700 1 -- 172.21.15.67:0/2477849175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c063730 msgr2=0x7efe5c0c9af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.362+0000 7efe6229f700 1 --2- 172.21.15.67:0/2477849175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c063730 0x7efe5c0c9af0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7efe5c06d3b0 tx=0x7efe54009780 comp rx=0 tx=0).stop 2023-06-23T01:24:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.362+0000 7efe6229f700 1 -- 172.21.15.67:0/2477849175 shutdown_connections 2023-06-23T01:24:08.367 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.362+0000 7efe6229f700 1 --2- 172.21.15.67:0/2477849175 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c063730 0x7efe5c0c9af0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.362+0000 7efe6229f700 1 --2- 172.21.15.67:0/2477849175 >> v2:172.21.15.193:3300/0 conn(0x7efe5c1599f0 0x7efe5c06d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.362+0000 7efe6229f700 1 -- 172.21.15.67:0/2477849175 >> 172.21.15.67:0/2477849175 conn(0x7efe5c067420 msgr2=0x7efe5c067820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:08.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.362+0000 7efe6229f700 1 -- 172.21.15.67:0/2477849175 shutdown_connections 2023-06-23T01:24:08.370 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.362+0000 7efe6229f700 1 -- 172.21.15.67:0/2477849175 wait complete. 2023-06-23T01:24:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.363+0000 7efe6229f700 1 Processor -- start 2023-06-23T01:24:08.372 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.363+0000 7efe6229f700 1 -- start start 2023-06-23T01:24:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe6229f700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe5c063730 0x7efe5c0c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe6229f700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c06dee0 0x7efe5c1581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe6229f700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe5c1599f0 0x7efe5c169fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe6229f700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c16b280 con 0x7efe5c06dee0 2023-06-23T01:24:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe6229f700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c158710 con 0x7efe5c063730 2023-06-23T01:24:08.373 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe6229f700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c0b09d0 con 0x7efe5c1599f0 2023-06-23T01:24:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c06dee0 0x7efe5c1581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c06dee0 0x7efe5c1581d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53456/0 (socket says 172.21.15.67:53456) 2023-06-23T01:24:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 -- 172.21.15.67:0/3635561693 learned_addr learned my addr 172.21.15.67:0/3635561693 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5bfff700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe5c063730 0x7efe5c0c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe6083c700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe5c1599f0 0x7efe5c169fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 -- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe5c1599f0 msgr2=0x7efe5c169fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.374 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe5c1599f0 0x7efe5c169fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 -- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe5c063730 msgr2=0x7efe5c0c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe5c063730 0x7efe5c0c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 -- 172.21.15.67:0/3635561693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe54009a20 con 0x7efe5c06dee0 2023-06-23T01:24:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe5b7fe700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c06dee0 0x7efe5c1581d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00c810 tx=0x7efe4c007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe53fff700 1 -- 172.21.15.67:0/3635561693 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4c005b30 con 0x7efe5c06dee0 2023-06-23T01:24:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe53fff700 1 -- 172.21.15.67:0/3635561693 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe4c005c90 con 0x7efe5c06dee0 2023-06-23T01:24:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.364+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c16a9e0 con 0x7efe5c06dee0 2023-06-23T01:24:08.375 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.365+0000 7efe53fff700 1 -- 172.21.15.67:0/3635561693 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe4c007300 con 0x7efe5c06dee0 2023-06-23T01:24:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.365+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c16afa0 con 0x7efe5c06dee0 2023-06-23T01:24:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.365+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5c0b07a0 con 0x7efe5c06dee0 2023-06-23T01:24:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.366+0000 7efe53fff700 1 -- 172.21.15.67:0/3635561693 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7efe4c0076b0 con 0x7efe5c06dee0 2023-06-23T01:24:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.366+0000 7efe53fff700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7efe3c066e90 0x7efe3c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.366+0000 7efe53fff700 1 -- 172.21.15.67:0/3635561693 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7efe4c0837e0 con 0x7efe5c06dee0 2023-06-23T01:24:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.366+0000 7efe5bfff700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7efe3c066e90 0x7efe3c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.376 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.367+0000 7efe5bfff700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7efe3c066e90 0x7efe3c069340 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efe44006fd0 tx=0x7efe44008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.377 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.371+0000 7efe53fff700 1 -- 172.21.15.67:0/3635561693 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7efe5c0b07a0 con 0x7efe5c06dee0 2023-06-23T01:24:08.566 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.563+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efe5c0b07a0 con 0x7efe5c06dee0 2023-06-23T01:24:08.566 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.564+0000 7efe53fff700 1 -- 172.21.15.67:0/3635561693 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7efe4c017020 con 0x7efe5c06dee0 2023-06-23T01:24:08.567 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2023-06-23T01:24:08.568 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7efe3c066e90 msgr2=0x7efe3c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.568 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7efe3c066e90 0x7efe3c069340 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7efe44006fd0 tx=0x7efe44008040 comp rx=0 tx=0).stop 2023-06-23T01:24:08.569 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c06dee0 msgr2=0x7efe5c1581d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.569 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c06dee0 0x7efe5c1581d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7efe4c00c810 tx=0x7efe4c007df0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.569 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 shutdown_connections 2023-06-23T01:24:08.569 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7efe3c066e90 0x7efe3c069340 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7efe5c1599f0 0x7efe5c169fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efe5c063730 0x7efe5c0c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 --2- 172.21.15.67:0/3635561693 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7efe5c06dee0 0x7efe5c1581d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 >> 172.21.15.67:0/3635561693 conn(0x7efe5c067420 msgr2=0x7efe5c158f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:08.570 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 shutdown_connections 2023-06-23T01:24:08.571 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.566+0000 7efe6229f700 1 -- 172.21.15.67:0/3635561693 wait complete. 2023-06-23T01:24:08.580 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T01:24:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.700+0000 7f62071b9700 1 Processor -- start 2023-06-23T01:24:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.700+0000 7f62071b9700 1 -- start start 2023-06-23T01:24:08.706 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62071b9700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620006e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62071b9700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620015e820 con 0x7f62000d0b70 2023-06-23T01:24:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62071b9700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f620015d970 con 0x7f620006dd50 2023-06-23T01:24:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62071b9700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62000b55e0 con 0x7f6200159c20 2023-06-23T01:24:08.707 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62059b6700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f62000d0b70 0x7f6200159ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60022/0 (socket says 172.21.15.67:60022) 2023-06-23T01:24:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62059b6700 1 -- 172.21.15.67:0/4009996104 learned_addr learned my addr 172.21.15.67:0/4009996104 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62061b7700 1 --2- 172.21.15.67:0/4009996104 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620006e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.708 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62051b5700 1 -- 172.21.15.67:0/4009996104 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2490976176 0 0) 0x7f620015e820 con 0x7f62000d0b70 2023-06-23T01:24:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62051b5700 1 -- 172.21.15.67:0/4009996104 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61e8003570 con 0x7f62000d0b70 2023-06-23T01:24:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62051b5700 1 -- 172.21.15.67:0/4009996104 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960399169 0 0) 0x7f620015d970 con 0x7f620006dd50 2023-06-23T01:24:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62051b5700 1 -- 172.21.15.67:0/4009996104 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f620015e820 con 0x7f620006dd50 2023-06-23T01:24:08.709 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62061b7700 1 -- 172.21.15.67:0/4009996104 >> v1:172.21.15.193:6789/0 conn(0x7f620006dd50 legacy=0x7f620015a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62061b7700 1 -- 172.21.15.67:0/4009996104 >> v1:172.21.15.67:6789/0 conn(0x7f62000d0b70 legacy=0x7f6200159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.701+0000 7f62061b7700 1 -- 172.21.15.67:0/4009996104 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f620015dc10 con 0x7f6200159c20 2023-06-23T01:24:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62061b7700 1 --2- 172.21.15.67:0/4009996104 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620006e9e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009910 tx=0x7f61f00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26d07d70e875c0c7 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62051b5700 1 -- 172.21.15.67:0/4009996104 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0011630 con 0x7f6200159c20 2023-06-23T01:24:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62051b5700 1 -- 172.21.15.67:0/4009996104 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f0011790 con 0x7f6200159c20 2023-06-23T01:24:08.710 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62051b5700 1 -- 172.21.15.67:0/4009996104 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f0011b30 con 0x7f6200159c20 2023-06-23T01:24:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62071b9700 1 -- 172.21.15.67:0/4009996104 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 msgr2=0x7f620006e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62071b9700 1 --2- 172.21.15.67:0/4009996104 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620006e9e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009910 tx=0x7f61f00091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62071b9700 1 -- 172.21.15.67:0/4009996104 shutdown_connections 2023-06-23T01:24:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62071b9700 1 --2- 172.21.15.67:0/4009996104 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620006e9e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62071b9700 1 -- 172.21.15.67:0/4009996104 >> 172.21.15.67:0/4009996104 conn(0x7f6200063020 msgr2=0x7f6200063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62071b9700 1 -- 172.21.15.67:0/4009996104 shutdown_connections 2023-06-23T01:24:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.702+0000 7f62071b9700 1 -- 172.21.15.67:0/4009996104 wait complete. 2023-06-23T01:24:08.711 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.703+0000 7f62071b9700 1 Processor -- start 2023-06-23T01:24:08.712 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.703+0000 7f62071b9700 1 -- start start 2023-06-23T01:24:08.714 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62071b9700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f620006dd50 0x7f620014e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.714 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62071b9700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62000d0b70 0x7f620015f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.714 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62071b9700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620015b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.714 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62071b9700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620014cff0 con 0x7f62000d0b70 2023-06-23T01:24:08.714 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62071b9700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620014de50 con 0x7f620006dd50 2023-06-23T01:24:08.714 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62071b9700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f620014e820 con 0x7f6200159c20 2023-06-23T01:24:08.714 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62059b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62000d0b70 0x7f620015f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62059b6700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62000d0b70 0x7f620015f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53460/0 (socket says 172.21.15.67:53460) 2023-06-23T01:24:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62059b6700 1 -- 172.21.15.67:0/159048688 learned_addr learned my addr 172.21.15.67:0/159048688 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62061b7700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f620006dd50 0x7f620014e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62069b8700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620015b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62059b6700 1 -- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 msgr2=0x7f620015b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.704+0000 7f62059b6700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620015b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.715 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f62059b6700 1 -- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f620006dd50 msgr2=0x7f620014e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f62059b6700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f620006dd50 0x7f620014e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f62059b6700 1 -- 172.21.15.67:0/159048688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61f0009420 con 0x7f62000d0b70 2023-06-23T01:24:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f62061b7700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f620006dd50 0x7f620014e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:24:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f62059b6700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62000d0b70 0x7f620015f3c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f61f400c7c0 tx=0x7f61f4007ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f61fe7fc700 1 -- 172.21.15.67:0/159048688 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4005b30 con 0x7f62000d0b70 2023-06-23T01:24:08.716 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f61fe7fc700 1 -- 172.21.15.67:0/159048688 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61f4005c90 con 0x7f62000d0b70 2023-06-23T01:24:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f620014d240 con 0x7f62000d0b70 2023-06-23T01:24:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f61fe7fc700 1 -- 172.21.15.67:0/159048688 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61f4007300 con 0x7f62000d0b70 2023-06-23T01:24:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.705+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f620014d800 con 0x7f62000d0b70 2023-06-23T01:24:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.706+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f620005f060 con 0x7f62000d0b70 2023-06-23T01:24:08.717 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.706+0000 7f61fe7fc700 1 -- 172.21.15.67:0/159048688 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f61f401b3b0 con 0x7f62000d0b70 2023-06-23T01:24:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.707+0000 7f61fe7fc700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f61ec066e90 0x7f61ec069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.707+0000 7f61fe7fc700 1 -- 172.21.15.67:0/159048688 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f61f40837e0 con 0x7f62000d0b70 2023-06-23T01:24:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.707+0000 7f62061b7700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f61ec066e90 0x7f61ec069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.707+0000 7f62061b7700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f61ec066e90 0x7f61ec069340 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f61f00091e0 tx=0x7f61f0017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:08.718 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.711+0000 7f61fe7fc700 1 -- 172.21.15.67:0/159048688 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f61f4050d00 con 0x7f62000d0b70 2023-06-23T01:24:08.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.905+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f620014df90 con 0x7f62000d0b70 2023-06-23T01:24:08.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.905+0000 7f61fe7fc700 1 -- 172.21.15.67:0/159048688 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f61f40170a0 con 0x7f62000d0b70 2023-06-23T01:24:08.908 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 8 2023-06-23T01:24:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f61ec066e90 msgr2=0x7f61ec069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f61ec066e90 0x7f61ec069340 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f61f00091e0 tx=0x7f61f0017000 comp rx=0 tx=0).stop 2023-06-23T01:24:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62000d0b70 msgr2=0x7f620015f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62000d0b70 0x7f620015f3c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f61f400c7c0 tx=0x7f61f4007ea0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 shutdown_connections 2023-06-23T01:24:08.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f61ec066e90 0x7f61ec069340 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f6200159c20 0x7f620015b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f620006dd50 0x7f620014e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 --2- 172.21.15.67:0/159048688 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f62000d0b70 0x7f620015f3c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.907+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 >> 172.21.15.67:0/159048688 conn(0x7f6200063020 msgr2=0x7f62000cfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.908+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 shutdown_connections 2023-06-23T01:24:08.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:08.908+0000 7f62071b9700 1 -- 172.21.15.67:0/159048688 wait complete. 2023-06-23T01:24:08.921 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:24:09.041 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.039+0000 7f0719033700 1 Processor -- start 2023-06-23T01:24:09.042 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.039+0000 7f0719033700 1 -- start start 2023-06-23T01:24:09.048 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0719033700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0714159eb0 0x7f07140cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0719033700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f07140cb6e0 0x7f071415dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0719033700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07140cbb00 con 0x7f0714159ae0 2023-06-23T01:24:09.049 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0719033700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f071415e8f0 con 0x7f07140cb6e0 2023-06-23T01:24:09.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0719033700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07140c5c30 con 0x7f0714159eb0 2023-06-23T01:24:09.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f071259c700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0714159eb0 0x7f07140cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.050 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f071359e700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f07140cb6e0 0x7f071415dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0712d9d700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f0714159ae0 0x7f07140c69f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.67:38760/0 (socket says 172.21.15.67:38760) 2023-06-23T01:24:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0712d9d700 1 -- 172.21.15.67:0/140021536 learned_addr learned my addr 172.21.15.67:0/140021536 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0711d9b700 1 -- 172.21.15.67:0/140021536 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2207373713 0 0) 0x7f07140cbb00 con 0x7f0714159ae0 2023-06-23T01:24:09.051 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f0711d9b700 1 -- 172.21.15.67:0/140021536 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06fc003570 con 0x7f0714159ae0 2023-06-23T01:24:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f071259c700 1 -- 172.21.15.67:0/140021536 >> v2:172.21.15.193:3300/0 conn(0x7f07140cb6e0 msgr2=0x7f071415dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f071259c700 1 --2- 172.21.15.67:0/140021536 >> v2:172.21.15.193:3300/0 conn(0x7f07140cb6e0 0x7f071415dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f071259c700 1 -- 172.21.15.67:0/140021536 >> v1:172.21.15.193:6789/0 conn(0x7f0714159ae0 legacy=0x7f07140c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.052 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.040+0000 7f071259c700 1 -- 172.21.15.67:0/140021536 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07140c5f10 con 0x7f0714159eb0 2023-06-23T01:24:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f071359e700 1 --2- 172.21.15.67:0/140021536 >> v2:172.21.15.193:3300/0 conn(0x7f07140cb6e0 0x7f071415dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:24:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f071259c700 1 --2- 172.21.15.67:0/140021536 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0714159eb0 0x7f07140cb1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f07080098e0 tx=0x7f07080091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafe3658c34e5fca server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.053 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0711d9b700 1 -- 172.21.15.67:0/140021536 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0708011630 con 0x7f0714159eb0 2023-06-23T01:24:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0711d9b700 1 -- 172.21.15.67:0/140021536 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0708011790 con 0x7f0714159eb0 2023-06-23T01:24:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0711d9b700 1 -- 172.21.15.67:0/140021536 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0708011b30 con 0x7f0714159eb0 2023-06-23T01:24:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0719033700 1 -- 172.21.15.67:0/140021536 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0714159eb0 msgr2=0x7f07140cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.054 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0719033700 1 --2- 172.21.15.67:0/140021536 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0714159eb0 0x7f07140cb1a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f07080098e0 tx=0x7f07080091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0719033700 1 -- 172.21.15.67:0/140021536 shutdown_connections 2023-06-23T01:24:09.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0719033700 1 --2- 172.21.15.67:0/140021536 >> v2:172.21.15.193:3300/0 conn(0x7f07140cb6e0 0x7f071415dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.055 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0719033700 1 --2- 172.21.15.67:0/140021536 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0714159eb0 0x7f07140cb1a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0719033700 1 -- 172.21.15.67:0/140021536 >> 172.21.15.67:0/140021536 conn(0x7f0714063020 msgr2=0x7f0714063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0719033700 1 -- 172.21.15.67:0/140021536 shutdown_connections 2023-06-23T01:24:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.041+0000 7f0719033700 1 -- 172.21.15.67:0/140021536 wait complete. 2023-06-23T01:24:09.056 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.043+0000 7f0719033700 1 Processor -- start 2023-06-23T01:24:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.043+0000 7f0719033700 1 -- start start 2023-06-23T01:24:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.043+0000 7f0719033700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07140cb6e0 0x7f071416a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.043+0000 7f0719033700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0714159ae0 0x7f071415c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.057 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.043+0000 7f0719033700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0714159eb0 0x7f07140d0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.043+0000 7f0719033700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071416a9b0 con 0x7f0714159ae0 2023-06-23T01:24:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.043+0000 7f0719033700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f071415ccc0 con 0x7f0714159eb0 2023-06-23T01:24:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.043+0000 7f0719033700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0714156360 con 0x7f07140cb6e0 2023-06-23T01:24:09.058 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0714159ae0 0x7f071415c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0714159ae0 0x7f071415c780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53466/0 (socket says 172.21.15.67:53466) 2023-06-23T01:24:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 -- 172.21.15.67:0/526238587 learned_addr learned my addr 172.21.15.67:0/526238587 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f0712d9d700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07140cb6e0 0x7f071416a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.059 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071359e700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0714159eb0 0x7f07140d0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.060 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 -- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07140cb6e0 msgr2=0x7f071416a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.060 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07140cb6e0 0x7f071416a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.060 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 -- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0714159eb0 msgr2=0x7f07140d0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0714159eb0 0x7f07140d0e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 -- 172.21.15.67:0/526238587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c009a20 con 0x7f0714159ae0 2023-06-23T01:24:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071359e700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0714159eb0 0x7f07140d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:24:09.061 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f071259c700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0714159ae0 0x7f071415c780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0708009ed0 tx=0x7f0708017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f06faffd700 1 -- 172.21.15.67:0/526238587 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0708011b10 con 0x7f0714159ae0 2023-06-23T01:24:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f06faffd700 1 -- 172.21.15.67:0/526238587 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0708000cc0 con 0x7f0714159ae0 2023-06-23T01:24:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0708009420 con 0x7f0714159ae0 2023-06-23T01:24:09.062 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f06faffd700 1 -- 172.21.15.67:0/526238587 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0708025640 con 0x7f0714159ae0 2023-06-23T01:24:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.044+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f071416d450 con 0x7f0714159ae0 2023-06-23T01:24:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.045+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f071405f060 con 0x7f0714159ae0 2023-06-23T01:24:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.046+0000 7f06faffd700 1 -- 172.21.15.67:0/526238587 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f07080259f0 con 0x7f0714159ae0 2023-06-23T01:24:09.063 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.046+0000 7f06faffd700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0700066e90 0x7f0700069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.064 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.046+0000 7f06faffd700 1 -- 172.21.15.67:0/526238587 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0708085930 con 0x7f0714159ae0 2023-06-23T01:24:09.064 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.046+0000 7f0712d9d700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0700066e90 0x7f0700069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.047+0000 7f0712d9d700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0700066e90 0x7f0700069340 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0704007fd0 tx=0x7f0704009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.065 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.051+0000 7f06faffd700 1 -- 172.21.15.67:0/526238587 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f0708052e50 con 0x7f0714159ae0 2023-06-23T01:24:09.296 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.294+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0714155c80 con 0x7f0714159ae0 2023-06-23T01:24:09.297 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.294+0000 7f06faffd700 1 -- 172.21.15.67:0/526238587 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3527 (secure 0 0 0) 0x7f0708015de0 con 0x7f0714159ae0 2023-06-23T01:24:09.297 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:24:09.298 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4266,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6834/3608562905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6832","nonce":3608562905},{"type":"v1","addr":"172.21.15.67:6834","nonce":3608562905}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.67:6835/3974490624","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6833","nonce":3974490624},{"type":"v1","addr":"172.21.15.67:6835","nonce":3974490624}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4278,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/643718169","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":643718169},{"type":"v1","addr":"172.21.15.193:6837","nonce":643718169}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4291,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6836/3137999631","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6835","nonce":3137999631},{"type":"v1","addr":"172.21.15.193:6836","nonce":3137999631}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-23T01:24:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0700066e90 msgr2=0x7f0700069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0700066e90 0x7f0700069340 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0704007fd0 tx=0x7f0704009040 comp rx=0 tx=0).stop 2023-06-23T01:24:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0714159ae0 msgr2=0x7f071415c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0714159ae0 0x7f071415c780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0708009ed0 tx=0x7f0708017000 comp rx=0 tx=0).stop 2023-06-23T01:24:09.299 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 shutdown_connections 2023-06-23T01:24:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0700066e90 0x7f0700069340 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0714159eb0 0x7f07140d0e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f07140cb6e0 0x7f071416a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.296+0000 7f0719033700 1 --2- 172.21.15.67:0/526238587 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0714159ae0 0x7f071415c780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.297+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 >> 172.21.15.67:0/526238587 conn(0x7f0714063020 msgr2=0x7f071415db00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:09.300 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.297+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 shutdown_connections 2023-06-23T01:24:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.297+0000 7f0719033700 1 -- 172.21.15.67:0/526238587 wait complete. 2023-06-23T01:24:09.301 INFO:teuthology.orchestra.run.smithi067.stderr:dumped fsmap epoch 3 2023-06-23T01:24:09.311 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:24:09.430 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.428+0000 7f7ddd39c700 1 Processor -- start 2023-06-23T01:24:09.431 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.428+0000 7f7ddd39c700 1 -- start start 2023-06-23T01:24:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7ddd39c700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8159dc0 0x7f7dd80ab230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7ddd39c700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7dd8069250 0x7f7dd80c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7ddd39c700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dd80c99c0 con 0x7f7dd80ab770 2023-06-23T01:24:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7ddd39c700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd80ac490 con 0x7f7dd8069250 2023-06-23T01:24:09.432 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7ddd39c700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd80abbf0 con 0x7f7dd8159dc0 2023-06-23T01:24:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7dd77fe700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f7dd8069250 0x7f7dd80c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7dd6ffd700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8159dc0 0x7f7dd80ab230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.433 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7dd67fc700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f7dd80ab770 0x7f7dd8068b40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.67:38776/0 (socket says 172.21.15.67:38776) 2023-06-23T01:24:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7dd67fc700 1 -- 172.21.15.67:0/366670842 learned_addr learned my addr 172.21.15.67:0/366670842 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7dd6ffd700 1 -- 172.21.15.67:0/366670842 >> v2:172.21.15.193:3300/0 conn(0x7f7dd8069250 msgr2=0x7f7dd80c9e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7dd6ffd700 1 --2- 172.21.15.67:0/366670842 >> v2:172.21.15.193:3300/0 conn(0x7f7dd8069250 0x7f7dd80c9e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.434 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7dd6ffd700 1 -- 172.21.15.67:0/366670842 >> v1:172.21.15.193:6789/0 conn(0x7f7dd80ab770 legacy=0x7f7dd8068b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.429+0000 7f7dd6ffd700 1 -- 172.21.15.67:0/366670842 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd80abed0 con 0x7f7dd8159dc0 2023-06-23T01:24:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7dd6ffd700 1 --2- 172.21.15.67:0/366670842 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8159dc0 0x7f7dd80ab230 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00098d0 tx=0x7f7dd0009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a52ecdf932f0ab9a server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7dd5ffb700 1 -- 172.21.15.67:0/366670842 <== mon.2 v2:172.21.15.193:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0005c10 con 0x7f7dd8159dc0 2023-06-23T01:24:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7dd5ffb700 1 -- 172.21.15.67:0/366670842 <== mon.2 v2:172.21.15.193:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd0005d70 con 0x7f7dd8159dc0 2023-06-23T01:24:09.435 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7dd5ffb700 1 -- 172.21.15.67:0/366670842 <== mon.2 v2:172.21.15.193:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dd0006280 con 0x7f7dd8159dc0 2023-06-23T01:24:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7ddd39c700 1 -- 172.21.15.67:0/366670842 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8159dc0 msgr2=0x7f7dd80ab230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/366670842 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8159dc0 0x7f7dd80ab230 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f7dd00098d0 tx=0x7f7dd0009120 comp rx=0 tx=0).stop 2023-06-23T01:24:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7ddd39c700 1 -- 172.21.15.67:0/366670842 shutdown_connections 2023-06-23T01:24:09.440 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/366670842 >> v2:172.21.15.193:3300/0 conn(0x7f7dd8069250 0x7f7dd80c9e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/366670842 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8159dc0 0x7f7dd80ab230 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7ddd39c700 1 -- 172.21.15.67:0/366670842 >> 172.21.15.67:0/366670842 conn(0x7f7dd80cd970 msgr2=0x7f7dd8067420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7ddd39c700 1 -- 172.21.15.67:0/366670842 shutdown_connections 2023-06-23T01:24:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.430+0000 7f7ddd39c700 1 -- 172.21.15.67:0/366670842 wait complete. 2023-06-23T01:24:09.441 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7ddd39c700 1 Processor -- start 2023-06-23T01:24:09.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7ddd39c700 1 -- start start 2023-06-23T01:24:09.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7ddd39c700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8069250 0x7f7dd815ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7ddd39c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd8159dc0 0x7f7dd8149860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.442 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7ddd39c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dd80afcc0 0x7f7dd80cae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7ddd39c700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd8156e80 con 0x7f7dd8159dc0 2023-06-23T01:24:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7ddd39c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd815b160 con 0x7f7dd80afcc0 2023-06-23T01:24:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7ddd39c700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd80b01f0 con 0x7f7dd8069250 2023-06-23T01:24:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7dd67fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd8159dc0 0x7f7dd8149860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.443 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7dd67fc700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd8159dc0 0x7f7dd8149860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53480/0 (socket says 172.21.15.67:53480) 2023-06-23T01:24:09.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.432+0000 7f7dd67fc700 1 -- 172.21.15.67:0/424607948 learned_addr learned my addr 172.21.15.67:0/424607948 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:09.444 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dd6ffd700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8069250 0x7f7dd815ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dd77fe700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dd80afcc0 0x7f7dd80cae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dd67fc700 1 -- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8069250 msgr2=0x7f7dd815ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dd67fc700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8069250 0x7f7dd815ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dd67fc700 1 -- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dd80afcc0 msgr2=0x7f7dd80cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.445 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dd67fc700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dd80afcc0 0x7f7dd80cae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dd67fc700 1 -- 172.21.15.67:0/424607948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd00093c0 con 0x7f7dd8159dc0 2023-06-23T01:24:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dd67fc700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd8159dc0 0x7f7dd8149860 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000c7c0 tx=0x7f7dc0007f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dceffd700 1 -- 172.21.15.67:0/424607948 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0005b30 con 0x7f7dd8159dc0 2023-06-23T01:24:09.446 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dceffd700 1 -- 172.21.15.67:0/424607948 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc0005cb0 con 0x7f7dd8159dc0 2023-06-23T01:24:09.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd80b04d0 con 0x7f7dd8159dc0 2023-06-23T01:24:09.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7dceffd700 1 -- 172.21.15.67:0/424607948 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc0004a80 con 0x7f7dd8159dc0 2023-06-23T01:24:09.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.433+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd80b0a90 con 0x7f7dd8159dc0 2023-06-23T01:24:09.447 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.434+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd80b1230 con 0x7f7dd8159dc0 2023-06-23T01:24:09.448 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.434+0000 7f7dceffd700 1 -- 172.21.15.67:0/424607948 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f7dc00070f0 con 0x7f7dd8159dc0 2023-06-23T01:24:09.448 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.435+0000 7f7dceffd700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f7db8066e90 0x7f7db8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.448 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.435+0000 7f7dceffd700 1 -- 172.21.15.67:0/424607948 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f7dc00835e0 con 0x7f7dd8159dc0 2023-06-23T01:24:09.448 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.435+0000 7f7dd6ffd700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f7db8066e90 0x7f7db8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.448 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.435+0000 7f7dd6ffd700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f7db8066e90 0x7f7db8069340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0009160 tx=0x7f7dd0016000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.449 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.439+0000 7f7dceffd700 1 -- 172.21.15.67:0/424607948 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f7dd80b1230 con 0x7f7dd8159dc0 2023-06-23T01:24:09.637 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.635+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dd80b1230 con 0x7f7dd8159dc0 2023-06-23T01:24:09.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.636+0000 7f7dceffd700 1 -- 172.21.15.67:0/424607948 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+13695 (secure 0 0 0) 0x7f7dd80b1230 con 0x7f7dd8159dc0 2023-06-23T01:24:09.639 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:24:09.639 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":19,"fsid":"0068cd93-d7dd-4502-a85d-c8368ea7e9d2","created":"2023-06-23T01:23:54.450197+0000","modified":"2023-06-23T01:24:08.856441+0000","last_up_change":"2023-06-23T01:24:00.795797+0000","last_in_change":"2023-06-23T01:23:57.782500+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.69999998807907104,"backfillfull_ratio":0.60000002384185791,"nearfull_ratio":0.60000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:24:01.045792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:24:06.629871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":111550},{"type":"v1","addr":"172.21.15.67:6801","nonce":111550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":111550},{"type":"v1","addr":"172.21.15.67:6803","nonce":111550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":111550},{"type":"v1","addr":"172.21.15.67:6807","nonce":111550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":111550},{"type":"v1","addr":"172.21.15.67:6805","nonce":111550}]},"public_addr":"172.21.15.67:6801/111550","cluster_addr":"172.21.15.67:6803/111550","heartbeat_back_addr":"172.21.15.67:6807/111550","heartbeat_front_addr":"172.21.15.67:6805/111550","state":["exists","up"]},{"osd":1,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":111554},{"type":"v1","addr":"172.21.15.67:6812","nonce":111554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":111554},{"type":"v1","addr":"172.21.15.67:6816","nonce":111554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":111554},{"type":"v1","addr":"172.21.15.67:6823","nonce":111554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":111554},{"type":"v1","addr":"172.21.15.67:6820","nonce":111554}]},"public_addr":"172.21.15.67:6812/111554","cluster_addr":"172.21.15.67:6816/111554","heartbeat_back_addr":"172.21.15.67:6823/111554","heartbeat_front_addr":"172.21.15.67:6820/111554","state":["exists","up"]},{"osd":2,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":111558},{"type":"v1","addr":"172.21.15.67:6809","nonce":111558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":111558},{"type":"v1","addr":"172.21.15.67:6813","nonce":111558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":111558},{"type":"v1","addr":"172.21.15.67:6821","nonce":111558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":111558},{"type":"v1","addr":"172.21.15.67:6817","nonce":111558}]},"public_addr":"172.21.15.67:6809/111558","cluster_addr":"172.21.15.67:6813/111558","heartbeat_back_addr":"172.21.15.67:6821/111558","heartbeat_front_addr":"172.21.15.67:6817/111558","state":["exists","up"]},{"osd":3,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":111567},{"type":"v1","addr":"172.21.15.67:6825","nonce":111567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":111567},{"type":"v1","addr":"172.21.15.67:6827","nonce":111567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":111567},{"type":"v1","addr":"172.21.15.67:6831","nonce":111567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":111567},{"type":"v1","addr":"172.21.15.67:6829","nonce":111567}]},"public_addr":"172.21.15.67:6825/111567","cluster_addr":"172.21.15.67:6827/111567","heartbeat_back_addr":"172.21.15.67:6831/111567","heartbeat_front_addr":"172.21.15.67:6829/111567","state":["exists","up"]},{"osd":4,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":83673},{"type":"v1","addr":"172.21.15.193:6801","nonce":83673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":83673},{"type":"v1","addr":"172.21.15.193:6803","nonce":83673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":83673},{"type":"v1","addr":"172.21.15.193:6807","nonce":83673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":83673},{"type":"v1","addr":"172.21.15.193:6805","nonce":83673}]},"public_addr":"172.21.15.193:6801/83673","cluster_addr":"172.21.15.193:6803/83673","heartbeat_back_addr":"172.21.15.193:6807/83673","heartbeat_front_addr":"172.21.15.193:6805/83673","state":["exists","up"]},{"osd":5,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83674},{"type":"v1","addr":"172.21.15.193:6809","nonce":83674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83674},{"type":"v1","addr":"172.21.15.193:6811","nonce":83674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83674},{"type":"v1","addr":"172.21.15.193:6815","nonce":83674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83674},{"type":"v1","addr":"172.21.15.193:6813","nonce":83674}]},"public_addr":"172.21.15.193:6809/83674","cluster_addr":"172.21.15.193:6811/83674","heartbeat_back_addr":"172.21.15.193:6815/83674","heartbeat_front_addr":"172.21.15.193:6813/83674","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":83699},{"type":"v1","addr":"172.21.15.193:6825","nonce":83699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":83699},{"type":"v1","addr":"172.21.15.193:6827","nonce":83699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":83699},{"type":"v1","addr":"172.21.15.193:6831","nonce":83699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":83699},{"type":"v1","addr":"172.21.15.193:6829","nonce":83699}]},"public_addr":"172.21.15.193:6825/83699","cluster_addr":"172.21.15.193:6827/83699","heartbeat_back_addr":"172.21.15.193:6831/83699","heartbeat_front_addr":"172.21.15.193:6829/83699","state":["exists","up"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":83675},{"type":"v1","addr":"172.21.15.193:6817","nonce":83675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":83675},{"type":"v1","addr":"172.21.15.193:6819","nonce":83675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":83675},{"type":"v1","addr":"172.21.15.193:6823","nonce":83675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":83675},{"type":"v1","addr":"172.21.15.193:6821","nonce":83675}]},"public_addr":"172.21.15.193:6817/83675","cluster_addr":"172.21.15.193:6819/83675","heartbeat_back_addr":"172.21.15.193:6823/83675","heartbeat_front_addr":"172.21.15.193:6821/83675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.993918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.906223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.928802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.986013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.876314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.922270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.983980+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.902644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:24:09.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f7db8066e90 msgr2=0x7f7db8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f7db8066e90 0x7f7db8069340 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7dd0009160 tx=0x7f7dd0016000 comp rx=0 tx=0).stop 2023-06-23T01:24:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd8159dc0 msgr2=0x7f7dd8149860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd8159dc0 0x7f7dd8149860 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f7dc000c7c0 tx=0x7f7dc0007f70 comp rx=0 tx=0).stop 2023-06-23T01:24:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 shutdown_connections 2023-06-23T01:24:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f7db8066e90 0x7f7db8069340 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7dd80afcc0 0x7f7dd80cae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f7dd8069250 0x7f7dd815ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 --2- 172.21.15.67:0/424607948 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f7dd8159dc0 0x7f7dd8149860 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 >> 172.21.15.67:0/424607948 conn(0x7f7dd80cd970 msgr2=0x7f7dd8157400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 shutdown_connections 2023-06-23T01:24:09.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.638+0000 7f7ddd39c700 1 -- 172.21.15.67:0/424607948 wait complete. 2023-06-23T01:24:09.652 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:24:01.045792+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-23T01:24:06.629871+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-06-23T01:24:09.653 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:24:09.774 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.772+0000 7f36e5c95700 1 Processor -- start 2023-06-23T01:24:09.775 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.772+0000 7f36e5c95700 1 -- start start 2023-06-23T01:24:09.779 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36e5c95700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f36e00cec80 0x7f36e015e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36e5c95700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e0067420 0x7f36e00677f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36e5c95700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36e015ac30 con 0x7f36e0157770 2023-06-23T01:24:09.780 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36e5c95700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36e015e7c0 con 0x7f36e00cec80 2023-06-23T01:24:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36e5c95700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e00b3d20 con 0x7f36e0067420 2023-06-23T01:24:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36e4c93700 1 --1- >> v1:172.21.15.193:6789/0 conn(0x7f36e0157770 0x7f36e00ce570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.67:38788/0 (socket says 172.21.15.67:38788) 2023-06-23T01:24:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36dffff700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f36e00cec80 0x7f36e015e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36e4c93700 1 -- 172.21.15.67:0/1847581678 learned_addr learned my addr 172.21.15.67:0/1847581678 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36e5494700 1 --2- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e0067420 0x7f36e00677f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.781 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.773+0000 7f36df7fe700 1 -- 172.21.15.67:0/1847581678 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999506451 0 0) 0x7f36e015ac30 con 0x7f36e0157770 2023-06-23T01:24:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36df7fe700 1 -- 172.21.15.67:0/1847581678 --> v1:172.21.15.193:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36c8003570 con 0x7f36e0157770 2023-06-23T01:24:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36dffff700 1 -- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e0067420 msgr2=0x7f36e00677f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36dffff700 1 --2- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e0067420 0x7f36e00677f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36dffff700 1 -- 172.21.15.67:0/1847581678 >> v1:172.21.15.193:6789/0 conn(0x7f36e0157770 legacy=0x7f36e00ce570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36dffff700 1 -- 172.21.15.67:0/1847581678 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e015a5e0 con 0x7f36e00cec80 2023-06-23T01:24:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36e5494700 1 --2- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e0067420 0x7f36e00677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:24:09.782 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36dffff700 1 --2- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e00cec80 0x7f36e015e280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f36d40098e0 tx=0x7f36d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26c915e89c56f9ca server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36df7fe700 1 -- 172.21.15.67:0/1847581678 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d4011630 con 0x7f36e00cec80 2023-06-23T01:24:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36df7fe700 1 -- 172.21.15.67:0/1847581678 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d4011790 con 0x7f36e00cec80 2023-06-23T01:24:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36df7fe700 1 -- 172.21.15.67:0/1847581678 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d4011b30 con 0x7f36e00cec80 2023-06-23T01:24:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36e5c95700 1 -- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e00cec80 msgr2=0x7f36e015e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36e5c95700 1 --2- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e00cec80 0x7f36e015e280 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f36d40098e0 tx=0x7f36d40091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36e5c95700 1 -- 172.21.15.67:0/1847581678 shutdown_connections 2023-06-23T01:24:09.783 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36e5c95700 1 --2- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e0067420 0x7f36e00677f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36e5c95700 1 --2- 172.21.15.67:0/1847581678 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e00cec80 0x7f36e015e280 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36e5c95700 1 -- 172.21.15.67:0/1847581678 >> 172.21.15.67:0/1847581678 conn(0x7f36e006ca00 msgr2=0x7f36e006ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:09.784 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.774+0000 7f36e5c95700 1 -- 172.21.15.67:0/1847581678 shutdown_connections 2023-06-23T01:24:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.775+0000 7f36e5c95700 1 -- 172.21.15.67:0/1847581678 wait complete. 2023-06-23T01:24:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.775+0000 7f36e5c95700 1 Processor -- start 2023-06-23T01:24:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.776+0000 7f36e5c95700 1 -- start start 2023-06-23T01:24:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.776+0000 7f36e5c95700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36e0067420 0x7f36e00c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.776+0000 7f36e5c95700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e00cec80 0x7f36e015dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.776+0000 7f36e5c95700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e0157770 0x7f36e0148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.785 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.776+0000 7f36e5c95700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e01559b0 con 0x7f36e0067420 2023-06-23T01:24:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.776+0000 7f36e5c95700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e00c9470 con 0x7f36e0157770 2023-06-23T01:24:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.776+0000 7f36e5c95700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36e0148ab0 con 0x7f36e00cec80 2023-06-23T01:24:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36e0067420 0x7f36e00c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36e0067420 0x7f36e00c85a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53486/0 (socket says 172.21.15.67:53486) 2023-06-23T01:24:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 -- 172.21.15.67:0/3239169036 learned_addr learned my addr 172.21.15.67:0/3239169036 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:09.786 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e5494700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e0157770 0x7f36e0148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36dffff700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e00cec80 0x7f36e015dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 -- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e00cec80 msgr2=0x7f36e015dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e00cec80 0x7f36e015dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 -- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e0157770 msgr2=0x7f36e0148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e0157770 0x7f36e0148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.787 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 -- 172.21.15.67:0/3239169036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d8009a20 con 0x7f36e0067420 2023-06-23T01:24:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e4c93700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36e0067420 0x7f36e00c85a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f36d000c5b0 tx=0x7f36d0007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36dcff9700 1 -- 172.21.15.67:0/3239169036 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0005b80 con 0x7f36e0067420 2023-06-23T01:24:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36dcff9700 1 -- 172.21.15.67:0/3239169036 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36d0005ce0 con 0x7f36e0067420 2023-06-23T01:24:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d4009420 con 0x7f36e0067420 2023-06-23T01:24:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36dcff9700 1 -- 172.21.15.67:0/3239169036 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d00072e0 con 0x7f36e0067420 2023-06-23T01:24:09.788 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.777+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e00c8bd0 con 0x7f36e0067420 2023-06-23T01:24:09.789 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.778+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e015b8d0 con 0x7f36e0067420 2023-06-23T01:24:09.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.779+0000 7f36dcff9700 1 -- 172.21.15.67:0/3239169036 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f36d0007690 con 0x7f36e0067420 2023-06-23T01:24:09.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.779+0000 7f36dcff9700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f36cc066e90 0x7f36cc069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:09.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.779+0000 7f36dcff9700 1 -- 172.21.15.67:0/3239169036 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f36d0083580 con 0x7f36e0067420 2023-06-23T01:24:09.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.779+0000 7f36dffff700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f36cc066e90 0x7f36cc069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:09.790 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.780+0000 7f36dffff700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f36cc066e90 0x7f36cc069340 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f36e00cf080 tx=0x7f36d4017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:09.791 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.783+0000 7f36dcff9700 1 -- 172.21.15.67:0/3239169036 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f36d0050c10 con 0x7f36e0067420 2023-06-23T01:24:09.978 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.976+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f36e005f060 con 0x7f36e0067420 2023-06-23T01:24:09.979 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.977+0000 7f36dcff9700 1 -- 172.21.15.67:0/3239169036 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f36d0017020 con 0x7f36e0067420 2023-06-23T01:24:09.980 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 1 2023-06-23T01:24:09.981 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f36cc066e90 msgr2=0x7f36cc069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f36cc066e90 0x7f36cc069340 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f36e00cf080 tx=0x7f36d4017000 comp rx=0 tx=0).stop 2023-06-23T01:24:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36e0067420 msgr2=0x7f36e00c85a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36e0067420 0x7f36e00c85a0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f36d000c5b0 tx=0x7f36d0007e00 comp rx=0 tx=0).stop 2023-06-23T01:24:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 shutdown_connections 2023-06-23T01:24:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f36cc066e90 0x7f36cc069340 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f36e0157770 0x7f36e0148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.982 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f36e0067420 0x7f36e00c85a0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 --2- 172.21.15.67:0/3239169036 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f36e00cec80 0x7f36e015dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 >> 172.21.15.67:0/3239169036 conn(0x7f36e006ca00 msgr2=0x7f36e0157c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 shutdown_connections 2023-06-23T01:24:09.983 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:09.979+0000 7f36e5c95700 1 -- 172.21.15.67:0/3239169036 wait complete. 2023-06-23T01:24:09.994 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T01:24:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.115+0000 7f1ba1816700 1 Processor -- start 2023-06-23T01:24:10.117 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.115+0000 7f1ba1816700 1 -- start start 2023-06-23T01:24:10.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.115+0000 7f1ba1816700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b9c159ae0 0x7f1b9c0c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.115+0000 7f1ba1816700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c0cb8b0 0x7f1b9c15dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.115+0000 7f1ba1816700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b9c0c5c30 con 0x7f1b9c159eb0 2023-06-23T01:24:10.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.115+0000 7f1ba1816700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9c15e8f0 con 0x7f1b9c159ae0 2023-06-23T01:24:10.119 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.115+0000 7f1ba1816700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0c5f10 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9affd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b9c159ae0 0x7f1b9c0c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9affd700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f1b9c159ae0 0x7f1b9c0c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53488/0 (socket says 172.21.15.67:53488) 2023-06-23T01:24:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9affd700 1 -- 172.21.15.67:0/2713274988 learned_addr learned my addr 172.21.15.67:0/2713274988 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9b7fe700 1 --2- 172.21.15.67:0/2713274988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c0cb8b0 0x7f1b9c15dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.120 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9affd700 1 -- 172.21.15.67:0/2713274988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c0cb8b0 msgr2=0x7f1b9c15dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9affd700 1 --2- 172.21.15.67:0/2713274988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c0cb8b0 0x7f1b9c15dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9affd700 1 -- 172.21.15.67:0/2713274988 >> v1:172.21.15.193:6789/0 conn(0x7f1b9c159eb0 legacy=0x7f1b9c0cb1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:10.121 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9affd700 1 -- 172.21.15.67:0/2713274988 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c0c61f0 con 0x7f1b9c159ae0 2023-06-23T01:24:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9b7fe700 1 --2- 172.21.15.67:0/2713274988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c0cb8b0 0x7f1b9c15dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:24:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b9affd700 1 --2- 172.21.15.67:0/2713274988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c159ae0 0x7f1b9c0c69f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1b940099b0 tx=0x7f1b94009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484b96e79413662c server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b99ffb700 1 -- 172.21.15.67:0/2713274988 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9400b820 con 0x7f1b9c159ae0 2023-06-23T01:24:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b99ffb700 1 -- 172.21.15.67:0/2713274988 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9400b980 con 0x7f1b9c159ae0 2023-06-23T01:24:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.116+0000 7f1b99ffb700 1 -- 172.21.15.67:0/2713274988 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9400bd20 con 0x7f1b9c159ae0 2023-06-23T01:24:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.117+0000 7f1ba1816700 1 -- 172.21.15.67:0/2713274988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c159ae0 msgr2=0x7f1b9c0c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.117+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2713274988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c159ae0 0x7f1b9c0c69f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1b940099b0 tx=0x7f1b94009250 comp rx=0 tx=0).stop 2023-06-23T01:24:10.122 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.117+0000 7f1ba1816700 1 -- 172.21.15.67:0/2713274988 shutdown_connections 2023-06-23T01:24:10.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.117+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2713274988 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c0cb8b0 0x7f1b9c15dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.117+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2713274988 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c159ae0 0x7f1b9c0c69f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.117+0000 7f1ba1816700 1 -- 172.21.15.67:0/2713274988 >> 172.21.15.67:0/2713274988 conn(0x7f1b9c063020 msgr2=0x7f1b9c063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:10.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.117+0000 7f1ba1816700 1 -- 172.21.15.67:0/2713274988 shutdown_connections 2023-06-23T01:24:10.123 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.117+0000 7f1ba1816700 1 -- 172.21.15.67:0/2713274988 wait complete. 2023-06-23T01:24:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.118+0000 7f1ba1816700 1 Processor -- start 2023-06-23T01:24:10.125 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.118+0000 7f1ba1816700 1 -- start start 2023-06-23T01:24:10.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1ba1816700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c0cb8b0 0x7f1b9c156ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1ba1816700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b9c159ae0 0x7f1b9c0ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1ba1816700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c159eb0 0x7f1b9c16a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1ba1816700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c16b320 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1ba1816700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c0c8da0 con 0x7f1b9c159ae0 2023-06-23T01:24:10.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1ba1816700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9c16ad60 con 0x7f1b9c159eb0 2023-06-23T01:24:10.126 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c0cb8b0 0x7f1b9c156ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c0cb8b0 0x7f1b9c156ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53504/0 (socket says 172.21.15.67:53504) 2023-06-23T01:24:10.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 -- 172.21.15.67:0/2672863269 learned_addr learned my addr 172.21.15.67:0/2672863269 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:10.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9a7fc700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b9c159ae0 0x7f1b9c0ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9b7fe700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c159eb0 0x7f1b9c16a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 -- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c159eb0 msgr2=0x7f1b9c16a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c159eb0 0x7f1b9c16a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 -- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b9c159ae0 msgr2=0x7f1b9c0ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.127 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b9c159ae0 0x7f1b9c0ceb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 -- 172.21.15.67:0/2672863269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b940094f0 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b9affd700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c0cb8b0 0x7f1b9c156ab0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009290 tx=0x7f1b94016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.119+0000 7f1b92ffd700 1 -- 172.21.15.67:0/2672863269 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b94011e20 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.120+0000 7f1b92ffd700 1 -- 172.21.15.67:0/2672863269 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9401b470 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.120+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b9c16b040 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.120+0000 7f1b92ffd700 1 -- 172.21.15.67:0/2672863269 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9401b830 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.120+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b9c156ff0 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.128 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.120+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b9c0c8890 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.121+0000 7f1b92ffd700 1 -- 172.21.15.67:0/2672863269 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f1b9401bbe0 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.121+0000 7f1b92ffd700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f1b7c066e90 0x7f1b7c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.121+0000 7f1b92ffd700 1 -- 172.21.15.67:0/2672863269 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f1b94085db0 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.122+0000 7f1b9a7fc700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f1b7c066e90 0x7f1b7c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.129 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.122+0000 7f1b9a7fc700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f1b7c066e90 0x7f1b7c069340 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c159450 tx=0x7f1b84009060 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.130 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.126+0000 7f1b92ffd700 1 -- 172.21.15.67:0/2672863269 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f1b9c0c8890 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.320 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.318+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1b9c0c8890 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.321 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.318+0000 7f1b92ffd700 1 -- 172.21.15.67:0/2672863269 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1b9402a020 con 0x7f1b9c0cb8b0 2023-06-23T01:24:10.321 INFO:teuthology.orchestra.run.smithi067.stdout:pg_num: 8 2023-06-23T01:24:10.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f1b7c066e90 msgr2=0x7f1b7c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f1b7c066e90 0x7f1b7c069340 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1b9c159450 tx=0x7f1b84009060 comp rx=0 tx=0).stop 2023-06-23T01:24:10.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c0cb8b0 msgr2=0x7f1b9c156ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c0cb8b0 0x7f1b9c156ab0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009290 tx=0x7f1b94016000 comp rx=0 tx=0).stop 2023-06-23T01:24:10.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 shutdown_connections 2023-06-23T01:24:10.323 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f1b7c066e90 0x7f1b7c069340 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f1b9c159eb0 0x7f1b9c16a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f1b9c0cb8b0 0x7f1b9c156ab0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 --2- 172.21.15.67:0/2672863269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b9c159ae0 0x7f1b9c0ceb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 >> 172.21.15.67:0/2672863269 conn(0x7f1b9c063020 msgr2=0x7f1b9c06db70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:10.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 shutdown_connections 2023-06-23T01:24:10.324 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.320+0000 7f1ba1816700 1 -- 172.21.15.67:0/2672863269 wait complete. 2023-06-23T01:24:10.335 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-23T01:24:10.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.453+0000 7f3c0f59e700 1 Processor -- start 2023-06-23T01:24:10.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.453+0000 7f3c0f59e700 1 -- start start 2023-06-23T01:24:10.457 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0f59e700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c1015b000 0x7f3c100d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0f59e700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3c100d10b0 0x7f3c1006cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0f59e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c100d1510 con 0x7f3c10159c20 2023-06-23T01:24:10.458 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0f59e700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c100b55e0 con 0x7f3c100d10b0 2023-06-23T01:24:10.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0f59e700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1015e820 con 0x7f3c1015b000 2023-06-23T01:24:10.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0ed9d700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3c100d10b0 0x7f3c1006cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0dd9b700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c1015b000 0x7f3c100d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0ed9d700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f3c100d10b0 0x7f3c1006cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.67:44968/0 (socket says 172.21.15.67:44968) 2023-06-23T01:24:10.459 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0dd9b700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c1015b000 0x7f3c100d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3301/0 says I am v2:172.21.15.67:57594/0 (socket says 172.21.15.67:57594) 2023-06-23T01:24:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0ed9d700 1 -- 172.21.15.67:0/703815937 learned_addr learned my addr 172.21.15.67:0/703815937 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.454+0000 7f3c0e59c700 1 --1- 172.21.15.67:0/703815937 >> v1:172.21.15.193:6789/0 conn(0x7f3c10159c20 0x7f3c1006e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.193:6789/0 says I am v1:172.21.15.67:38814/0 (socket says 172.21.15.67:38814) 2023-06-23T01:24:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0ed9d700 1 -- 172.21.15.67:0/703815937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c1015b000 msgr2=0x7f3c100d0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0ed9d700 1 --2- 172.21.15.67:0/703815937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c1015b000 0x7f3c100d0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.460 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0ed9d700 1 -- 172.21.15.67:0/703815937 >> v1:172.21.15.193:6789/0 conn(0x7f3c10159c20 legacy=0x7f3c1006e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0ed9d700 1 -- 172.21.15.67:0/703815937 --> v2:172.21.15.193:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1015d970 con 0x7f3c100d10b0 2023-06-23T01:24:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0d59a700 1 -- 172.21.15.67:0/703815937 <== mon.1 v1:172.21.15.193:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775485781 0 0) 0x7f3c08001110 con 0x7f3c10159c20 2023-06-23T01:24:10.461 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0ed9d700 1 --2- 172.21.15.67:0/703815937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c100d10b0 0x7f3c1006cd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009f20 tx=0x7f3c00009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef3cae1ab4a5aab1 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.464 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0d59a700 1 -- 172.21.15.67:0/703815937 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0000e070 con 0x7f3c100d10b0 2023-06-23T01:24:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0d59a700 1 -- 172.21.15.67:0/703815937 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0000b9d0 con 0x7f3c100d10b0 2023-06-23T01:24:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0d59a700 1 -- 172.21.15.67:0/703815937 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c000092e0 con 0x7f3c100d10b0 2023-06-23T01:24:10.465 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0f59e700 1 -- 172.21.15.67:0/703815937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c100d10b0 msgr2=0x7f3c1006cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.455+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/703815937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c100d10b0 0x7f3c1006cd40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3c00009f20 tx=0x7f3c00009840 comp rx=0 tx=0).stop 2023-06-23T01:24:10.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.456+0000 7f3c0f59e700 1 -- 172.21.15.67:0/703815937 shutdown_connections 2023-06-23T01:24:10.466 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.456+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/703815937 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c100d10b0 0x7f3c1006cd40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.456+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/703815937 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c1015b000 0x7f3c100d0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.467 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.456+0000 7f3c0f59e700 1 -- 172.21.15.67:0/703815937 >> 172.21.15.67:0/703815937 conn(0x7f3c10063020 msgr2=0x7f3c10063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.456+0000 7f3c0f59e700 1 -- 172.21.15.67:0/703815937 shutdown_connections 2023-06-23T01:24:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.456+0000 7f3c0f59e700 1 -- 172.21.15.67:0/703815937 wait complete. 2023-06-23T01:24:10.468 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.457+0000 7f3c0f59e700 1 Processor -- start 2023-06-23T01:24:10.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.457+0000 7f3c0f59e700 1 -- start start 2023-06-23T01:24:10.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.457+0000 7f3c0f59e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c100d10b0 0x7f3c100c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.469 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.457+0000 7f3c0f59e700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c10159c20 0x7f3c101529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.457+0000 7f3c0f59e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1015b000 0x7f3c10156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.457+0000 7f3c0f59e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c10169bd0 con 0x7f3c100d10b0 2023-06-23T01:24:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.457+0000 7f3c0f59e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c10169e50 con 0x7f3c1015b000 2023-06-23T01:24:10.470 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.457+0000 7f3c0f59e700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c10157790 con 0x7f3c10159c20 2023-06-23T01:24:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c100d10b0 0x7f3c100c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c100d10b0 0x7f3c100c80f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53512/0 (socket says 172.21.15.67:53512) 2023-06-23T01:24:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 -- 172.21.15.67:0/42981935 learned_addr learned my addr 172.21.15.67:0/42981935 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0dd9b700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c10159c20 0x7f3c101529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.471 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0ed9d700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1015b000 0x7f3c10156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 -- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c10159c20 msgr2=0x7f3c101529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.472 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c10159c20 0x7f3c101529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 -- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1015b000 msgr2=0x7f3c10156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.473 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1015b000 0x7f3c10156f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 -- 172.21.15.67:0/42981935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c00009ae0 con 0x7f3c100d10b0 2023-06-23T01:24:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0e59c700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c100d10b0 0x7f3c100c80f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800c810 tx=0x7f3c08007c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c067fc700 1 -- 172.21.15.67:0/42981935 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08005a70 con 0x7f3c100d10b0 2023-06-23T01:24:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c067fc700 1 -- 172.21.15.67:0/42981935 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c08005bf0 con 0x7f3c100d10b0 2023-06-23T01:24:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c10157a70 con 0x7f3c100d10b0 2023-06-23T01:24:10.474 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c067fc700 1 -- 172.21.15.67:0/42981935 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c080070d0 con 0x7f3c100d10b0 2023-06-23T01:24:10.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.458+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c101574a0 con 0x7f3c100d10b0 2023-06-23T01:24:10.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.459+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1005f060 con 0x7f3c100d10b0 2023-06-23T01:24:10.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.460+0000 7f3c067fc700 1 -- 172.21.15.67:0/42981935 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f3c080074c0 con 0x7f3c100d10b0 2023-06-23T01:24:10.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.460+0000 7f3c067fc700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f3bf0066e90 0x7f3bf0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.475 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.460+0000 7f3c067fc700 1 -- 172.21.15.67:0/42981935 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3c080835a0 con 0x7f3c100d10b0 2023-06-23T01:24:10.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.460+0000 7f3c0dd9b700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f3bf0066e90 0x7f3bf0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.476 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.461+0000 7f3c0dd9b700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f3bf0066e90 0x7f3bf0069340 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3c10158700 tx=0x7f3bf80091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.477 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.464+0000 7f3c067fc700 1 -- 172.21.15.67:0/42981935 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f3c08050ac0 con 0x7f3c100d10b0 2023-06-23T01:24:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.660+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f3c100d1510 con 0x7f3c100d10b0 2023-06-23T01:24:10.663 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.661+0000 7f3c067fc700 1 -- 172.21.15.67:0/42981935 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7f3c08017090 con 0x7f3c100d10b0 2023-06-23T01:24:10.663 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:24:10.664 INFO:teuthology.orchestra.run.smithi067.stdout:[ 2023-06-23T01:24:10.664 INFO:teuthology.orchestra.run.smithi067.stdout: { 2023-06-23T01:24:10.664 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 1, 2023-06-23T01:24:10.665 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": ".mgr" 2023-06-23T01:24:10.665 INFO:teuthology.orchestra.run.smithi067.stdout: }, 2023-06-23T01:24:10.665 INFO:teuthology.orchestra.run.smithi067.stdout: { 2023-06-23T01:24:10.665 INFO:teuthology.orchestra.run.smithi067.stdout: "poolnum": 2, 2023-06-23T01:24:10.666 INFO:teuthology.orchestra.run.smithi067.stdout: "poolname": "rbd" 2023-06-23T01:24:10.666 INFO:teuthology.orchestra.run.smithi067.stdout: } 2023-06-23T01:24:10.666 INFO:teuthology.orchestra.run.smithi067.stdout:] 2023-06-23T01:24:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f3bf0066e90 msgr2=0x7f3bf0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f3bf0066e90 0x7f3bf0069340 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3c10158700 tx=0x7f3bf80091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c100d10b0 msgr2=0x7f3c100c80f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c100d10b0 0x7f3c100c80f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f3c0800c810 tx=0x7f3c08007c50 comp rx=0 tx=0).stop 2023-06-23T01:24:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 shutdown_connections 2023-06-23T01:24:10.667 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f3bf0066e90 0x7f3bf0069340 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c1015b000 0x7f3c10156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f3c100d10b0 0x7f3c100c80f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.668 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 --2- 172.21.15.67:0/42981935 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f3c10159c20 0x7f3c101529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 >> 172.21.15.67:0/42981935 conn(0x7f3c10063020 msgr2=0x7f3c1006e4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 shutdown_connections 2023-06-23T01:24:10.669 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.663+0000 7f3c0f59e700 1 -- 172.21.15.67:0/42981935 wait complete. 2023-06-23T01:24:10.677 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-23T01:24:10.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-23T01:24:10.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.797+0000 7f0a0fc38700 1 Processor -- start 2023-06-23T01:24:10.800 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.797+0000 7f0a0fc38700 1 -- start start 2023-06-23T01:24:10.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0fc38700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a08159d90 0x7f0a080c6d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0fc38700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0a080cb1a0 0x7f0a080cb5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0fc38700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a080c6890 con 0x7f0a080b2b70 2023-06-23T01:24:10.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0fc38700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a080b3700 con 0x7f0a08159d90 2023-06-23T01:24:10.802 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0fc38700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a080cbb30 con 0x7f0a080cb1a0 2023-06-23T01:24:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0d9d4700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a08159d90 0x7f0a080c6d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0d9d4700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f0a08159d90 0x7f0a080c6d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53526/0 (socket says 172.21.15.67:53526) 2023-06-23T01:24:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0d9d4700 1 -- 172.21.15.67:0/2773776646 learned_addr learned my addr 172.21.15.67:0/2773776646 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.798+0000 7f0a0e1d5700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0a080b2b70 0x7f0a080b2ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60034/0 (socket says 172.21.15.67:60034) 2023-06-23T01:24:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0d1d3700 1 --2- 172.21.15.67:0/2773776646 >> v2:172.21.15.193:3300/0 conn(0x7f0a080cb1a0 0x7f0a080cb5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0d9d4700 1 -- 172.21.15.67:0/2773776646 >> v2:172.21.15.193:3300/0 conn(0x7f0a080cb1a0 msgr2=0x7f0a080cb5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:10.803 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0d9d4700 1 --2- 172.21.15.67:0/2773776646 >> v2:172.21.15.193:3300/0 conn(0x7f0a080cb1a0 0x7f0a080cb5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0d9d4700 1 -- 172.21.15.67:0/2773776646 >> v1:172.21.15.67:6789/0 conn(0x7f0a080b2b70 legacy=0x7f0a080b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0d9d4700 1 -- 172.21.15.67:0/2773776646 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a080c5ca0 con 0x7f0a08159d90 2023-06-23T01:24:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0d9d4700 1 --2- 172.21.15.67:0/2773776646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a08159d90 0x7f0a080c6d20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0a080714c0 tx=0x7f0a00009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=91ff0317324e0af2 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0c9d2700 1 -- 172.21.15.67:0/2773776646 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a0000e070 con 0x7f0a08159d90 2023-06-23T01:24:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0c9d2700 1 -- 172.21.15.67:0/2773776646 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a0000ba50 con 0x7f0a08159d90 2023-06-23T01:24:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.799+0000 7f0a0c9d2700 1 -- 172.21.15.67:0/2773776646 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a000092e0 con 0x7f0a08159d90 2023-06-23T01:24:10.804 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.800+0000 7f0a0fc38700 1 -- 172.21.15.67:0/2773776646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a08159d90 msgr2=0x7f0a080c6d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.800+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/2773776646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a08159d90 0x7f0a080c6d20 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f0a080714c0 tx=0x7f0a00009840 comp rx=0 tx=0).stop 2023-06-23T01:24:10.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.800+0000 7f0a0fc38700 1 -- 172.21.15.67:0/2773776646 shutdown_connections 2023-06-23T01:24:10.805 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.800+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/2773776646 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a08159d90 0x7f0a080c6d20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.800+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/2773776646 >> v2:172.21.15.193:3300/0 conn(0x7f0a080cb1a0 0x7f0a080cb5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.800+0000 7f0a0fc38700 1 -- 172.21.15.67:0/2773776646 >> 172.21.15.67:0/2773776646 conn(0x7f0a08063020 msgr2=0x7f0a08063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:10.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.800+0000 7f0a0fc38700 1 -- 172.21.15.67:0/2773776646 shutdown_connections 2023-06-23T01:24:10.806 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.800+0000 7f0a0fc38700 1 -- 172.21.15.67:0/2773776646 wait complete. 2023-06-23T01:24:10.808 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.801+0000 7f0a0fc38700 1 Processor -- start 2023-06-23T01:24:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.801+0000 7f0a0fc38700 1 -- start start 2023-06-23T01:24:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.801+0000 7f0a0fc38700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0a080b2b70 0x7f0a0815d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.801+0000 7f0a0fc38700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a080cb1a0 0x7f0a08154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.801+0000 7f0a0fc38700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a08159d90 0x7f0a0815a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.809 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.801+0000 7f0a0fc38700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a08155010 con 0x7f0a080cb1a0 2023-06-23T01:24:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.801+0000 7f0a0fc38700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0815ec00 con 0x7f0a08159d90 2023-06-23T01:24:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.801+0000 7f0a0fc38700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0815dff0 con 0x7f0a080b2b70 2023-06-23T01:24:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a080cb1a0 0x7f0a08154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a080cb1a0 0x7f0a08154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53538/0 (socket says 172.21.15.67:53538) 2023-06-23T01:24:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 -- 172.21.15.67:0/591561610 learned_addr learned my addr 172.21.15.67:0/591561610 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d9d4700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0a080b2b70 0x7f0a0815d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0e1d5700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a08159d90 0x7f0a0815a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.810 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 -- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0a080b2b70 msgr2=0x7f0a0815d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0a080b2b70 0x7f0a0815d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 -- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a08159d90 msgr2=0x7f0a0815a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a08159d90 0x7f0a0815a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 -- 172.21.15.67:0/591561610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a00009ae0 con 0x7f0a080cb1a0 2023-06-23T01:24:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0e1d5700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a08159d90 0x7f0a0815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:24:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0d1d3700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a080cb1a0 0x7f0a08154ad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00d8c0 tx=0x7f09fc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a05ffb700 1 -- 172.21.15.67:0/591561610 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc0125f0 con 0x7f0a080cb1a0 2023-06-23T01:24:10.811 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a05ffb700 1 -- 172.21.15.67:0/591561610 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc012750 con 0x7f0a080cb1a0 2023-06-23T01:24:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0815e2d0 con 0x7f0a080cb1a0 2023-06-23T01:24:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a05ffb700 1 -- 172.21.15.67:0/591561610 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc012af0 con 0x7f0a080cb1a0 2023-06-23T01:24:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.802+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0815e890 con 0x7f0a080cb1a0 2023-06-23T01:24:10.812 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.803+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a0815d2a0 con 0x7f0a080cb1a0 2023-06-23T01:24:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.804+0000 7f0a05ffb700 1 -- 172.21.15.67:0/591561610 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f09fc01e460 con 0x7f0a080cb1a0 2023-06-23T01:24:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.804+0000 7f0a05ffb700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f09f4066e90 0x7f09f4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.804+0000 7f0a05ffb700 1 -- 172.21.15.67:0/591561610 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f09fc0848b0 con 0x7f0a080cb1a0 2023-06-23T01:24:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.804+0000 7f0a0d9d4700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f09f4066e90 0x7f09f4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:10.813 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.805+0000 7f0a0d9d4700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f09f4066e90 0x7f09f4069340 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009880 tx=0x7f0a00018000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:10.814 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:10.809+0000 7f0a05ffb700 1 -- 172.21.15.67:0/591561610 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f09fc051dd0 con 0x7f0a080cb1a0 2023-06-23T01:24:11.006 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.003+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0a0805f060 con 0x7f0a080cb1a0 2023-06-23T01:24:11.986 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.984+0000 7f0a05ffb700 1 -- 172.21.15.67:0/591561610 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f09fc012e60 con 0x7f0a080cb1a0 2023-06-23T01:24:11.992 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f09f4066e90 msgr2=0x7f09f4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:11.992 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f09f4066e90 0x7f09f4069340 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0a00009880 tx=0x7f0a00018000 comp rx=0 tx=0).stop 2023-06-23T01:24:11.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a080cb1a0 msgr2=0x7f0a08154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:11.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a080cb1a0 0x7f0a08154ad0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f09fc00d8c0 tx=0x7f09fc008040 comp rx=0 tx=0).stop 2023-06-23T01:24:11.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 shutdown_connections 2023-06-23T01:24:11.993 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f09f4066e90 0x7f09f4069340 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a08159d90 0x7f0a0815a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0a080b2b70 0x7f0a0815d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:11.994 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 --2- 172.21.15.67:0/591561610 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0a080cb1a0 0x7f0a08154ad0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:11.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 >> 172.21.15.67:0/591561610 conn(0x7f0a08063020 msgr2=0x7f0a080cd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:11.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 shutdown_connections 2023-06-23T01:24:11.995 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:11.989+0000 7f0a0fc38700 1 -- 172.21.15.67:0/591561610 wait complete. 2023-06-23T01:24:11.996 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_metadata' created 2023-06-23T01:24:12.005 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-23T01:24:12.136 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.134+0000 7f0e70ab4700 1 Processor -- start 2023-06-23T01:24:12.136 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.134+0000 7f0e70ab4700 1 -- start start 2023-06-23T01:24:12.139 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e70ab4700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f0e6c15e1a0 0x7f0e6c1570e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e70ab4700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c0b3680 con 0x7f0e6c15e6d0 2023-06-23T01:24:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e70ab4700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e6c0b2a70 con 0x7f0e6c0b5790 2023-06-23T01:24:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e70ab4700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e6c0b2d50 con 0x7f0e6c15e1a0 2023-06-23T01:24:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e69d9b700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f0e6c15e6d0 0x7f0e6c15da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60044/0 (socket says 172.21.15.67:60044) 2023-06-23T01:24:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e69d9b700 1 -- 172.21.15.67:0/1419502064 learned_addr learned my addr 172.21.15.67:0/1419502064 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6ad9d700 1 --2- 172.21.15.67:0/1419502064 >> v2:172.21.15.193:3300/0 conn(0x7f0e6c15e1a0 0x7f0e6c1570e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:12.140 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1186140689 0 0) 0x7f0e6c0b3680 con 0x7f0e6c15e6d0 2023-06-23T01:24:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e58003570 con 0x7f0e6c15e6d0 2023-06-23T01:24:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 <== mon.0 v1:172.21.15.67:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 384166299 0 0) 0x7f0e58003570 con 0x7f0e6c15e6d0 2023-06-23T01:24:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 >> v2:172.21.15.193:3300/0 conn(0x7f0e6c15e1a0 msgr2=0x7f0e6c1570e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 --2- 172.21.15.67:0/1419502064 >> v2:172.21.15.193:3300/0 conn(0x7f0e6c15e1a0 0x7f0e6c1570e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.141 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 >> v1:172.21.15.193:6789/0 conn(0x7f0e6c0b5790 legacy=0x7f0e6c069290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:12.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 --> v1:172.21.15.67:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c0b3030 con 0x7f0e6c15e6d0 2023-06-23T01:24:12.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 <== mon.0 v1:172.21.15.67:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 918208491 0 0) 0x7f0e54001470 con 0x7f0e6c15e6d0 2023-06-23T01:24:12.142 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 >> v1:172.21.15.67:6789/0 conn(0x7f0e6c15e6d0 legacy=0x7f0e6c15da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e58003ad0 0x7f0e58003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58004810 0x7f0e58004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58005210 0x7f0e5800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0b2d50 con 0x7f0e58005210 2023-06-23T01:24:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5800c130 con 0x7f0e58004810 2023-06-23T01:24:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.135+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5800c2f0 con 0x7f0e58003ad0 2023-06-23T01:24:12.143 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6ad9d700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58005210 0x7f0e5800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e69d9b700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58004810 0x7f0e58004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6a59c700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e58003ad0 0x7f0e58003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6ad9d700 1 -- 172.21.15.67:0/1419502064 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e58003ad0 msgr2=0x7f0e58003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6ad9d700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e58003ad0 0x7f0e58003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6ad9d700 1 -- 172.21.15.67:0/1419502064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58004810 msgr2=0x7f0e58004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6ad9d700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58004810 0x7f0e58004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6ad9d700 1 -- 172.21.15.67:0/1419502064 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e58003570 con 0x7f0e58005210 2023-06-23T01:24:12.144 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6ad9d700 1 -- 172.21.15.67:0/1419502064 reap_dead start 2023-06-23T01:24:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6ad9d700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58005210 0x7f0e5800ba60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0b3630 tx=0x7f0e5c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecc1ec452bf95fef server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c00ea60 con 0x7f0e58005210 2023-06-23T01:24:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c00f040 con 0x7f0e58005210 2023-06-23T01:24:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.136+0000 7f0e6959a700 1 -- 172.21.15.67:0/1419502064 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c00ee10 con 0x7f0e58005210 2023-06-23T01:24:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.137+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1419502064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58005210 msgr2=0x7f0e5800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.137+0000 7f0e70ab4700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58005210 0x7f0e5800ba60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0b3630 tx=0x7f0e5c00b040 comp rx=0 tx=0).stop 2023-06-23T01:24:12.145 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.137+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1419502064 shutdown_connections 2023-06-23T01:24:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.137+0000 7f0e70ab4700 1 --2- 172.21.15.67:0/1419502064 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e58005210 0x7f0e5800ba60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.137+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1419502064 >> 172.21.15.67:0/1419502064 conn(0x7f0e6c0d0e30 msgr2=0x7f0e6c0d1230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.137+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1419502064 shutdown_connections 2023-06-23T01:24:12.146 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.137+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1419502064 wait complete. 2023-06-23T01:24:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.138+0000 7f0e70ab4700 1 Processor -- start 2023-06-23T01:24:12.148 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.138+0000 7f0e70ab4700 1 -- start start 2023-06-23T01:24:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e70ab4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58005210 0x7f0e6c0afe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e70ab4700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e6c0b03d0 0x7f0e6c068a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e70ab4700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6c068f60 0x7f0e6c16d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e70ab4700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c069400 con 0x7f0e6c068f60 2023-06-23T01:24:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e70ab4700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c0ad860 con 0x7f0e58005210 2023-06-23T01:24:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e70ab4700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e6c16e2b0 con 0x7f0e6c0b03d0 2023-06-23T01:24:12.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6c068f60 0x7f0e6c16d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6c068f60 0x7f0e6c16d5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53558/0 (socket says 172.21.15.67:53558) 2023-06-23T01:24:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 -- 172.21.15.67:0/1740492246 learned_addr learned my addr 172.21.15.67:0/1740492246 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6a59c700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58005210 0x7f0e6c0afe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e69d9b700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e6c0b03d0 0x7f0e6c068a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 -- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e6c0b03d0 msgr2=0x7f0e6c068a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e6c0b03d0 0x7f0e6c068a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 -- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58005210 msgr2=0x7f0e6c0afe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:12.150 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58005210 0x7f0e6c0afe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 -- 172.21.15.67:0/1740492246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c0ad210 con 0x7f0e6c068f60 2023-06-23T01:24:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e6ad9d700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6c068f60 0x7f0e6c16d5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0b1c30 tx=0x7f0e5c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e627fc700 1 -- 172.21.15.67:0/1740492246 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c00eee0 con 0x7f0e6c068f60 2023-06-23T01:24:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e627fc700 1 -- 172.21.15.67:0/1740492246 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5c022070 con 0x7f0e6c068f60 2023-06-23T01:24:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e6c0ad4f0 con 0x7f0e6c068f60 2023-06-23T01:24:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e627fc700 1 -- 172.21.15.67:0/1740492246 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5c01d890 con 0x7f0e6c068f60 2023-06-23T01:24:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.139+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e6c16dd40 con 0x7f0e6c068f60 2023-06-23T01:24:12.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.140+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e6c0ada10 con 0x7f0e6c068f60 2023-06-23T01:24:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.141+0000 7f0e627fc700 1 -- 172.21.15.67:0/1740492246 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f0e5c003020 con 0x7f0e6c068f60 2023-06-23T01:24:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.141+0000 7f0e627fc700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0e4c066e90 0x7f0e4c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.141+0000 7f0e627fc700 1 -- 172.21.15.67:0/1740492246 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f0e5c024070 con 0x7f0e6c068f60 2023-06-23T01:24:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.141+0000 7f0e6a59c700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0e4c066e90 0x7f0e4c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.142+0000 7f0e6a59c700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0e4c066e90 0x7f0e4c069340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e64007e40 tx=0x7f0e64007d30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:12.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.146+0000 7f0e627fc700 1 -- 172.21.15.67:0/1740492246 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f0e5c053810 con 0x7f0e6c068f60 2023-06-23T01:24:12.340 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.338+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0e6c05f060 con 0x7f0e6c068f60 2023-06-23T01:24:12.991 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.988+0000 7f0e627fc700 1 -- 172.21.15.67:0/1740492246 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v21) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0e5c056e30 con 0x7f0e6c068f60 2023-06-23T01:24:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.993+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0e4c066e90 msgr2=0x7f0e4c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.993+0000 7f0e70ab4700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0e4c066e90 0x7f0e4c069340 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0e64007e40 tx=0x7f0e64007d30 comp rx=0 tx=0).stop 2023-06-23T01:24:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.993+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6c068f60 msgr2=0x7f0e6c16d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:12.996 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.993+0000 7f0e70ab4700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6c068f60 0x7f0e6c16d5c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c0b1c30 tx=0x7f0e5c019000 comp rx=0 tx=0).stop 2023-06-23T01:24:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.994+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 shutdown_connections 2023-06-23T01:24:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.994+0000 7f0e70ab4700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f0e4c066e90 0x7f0e4c069340 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.994+0000 7f0e70ab4700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f0e6c068f60 0x7f0e6c16d5c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.997 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.994+0000 7f0e70ab4700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0e58005210 0x7f0e6c0afe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.994+0000 7f0e70ab4700 1 --2- 172.21.15.67:0/1740492246 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f0e6c0b03d0 0x7f0e6c068a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:12.998 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:12.994+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 >> 172.21.15.67:0/1740492246 conn(0x7f0e6c0d0e30 msgr2=0x7f0e6c06d930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:13.004 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.001+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 shutdown_connections 2023-06-23T01:24:13.005 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.003+0000 7f0e70ab4700 1 -- 172.21.15.67:0/1740492246 wait complete. 2023-06-23T01:24:13.007 INFO:teuthology.orchestra.run.smithi067.stderr:pool 'cephfs_data' created 2023-06-23T01:24:13.021 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-23T01:24:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.147+0000 7f5751f9e700 1 Processor -- start 2023-06-23T01:24:13.149 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.147+0000 7f5751f9e700 1 -- start start 2023-06-23T01:24:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f5751f9e700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f574c15b000 0x7f574c0d0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f5751f9e700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f574c0d10b0 0x7f574c06cd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.151 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f5751f9e700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f574c0d1510 con 0x7f574c159c20 2023-06-23T01:24:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f5751f9e700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c0b55e0 con 0x7f574c15b000 2023-06-23T01:24:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f5751f9e700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574c15e820 con 0x7f574c0d10b0 2023-06-23T01:24:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574bfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f574c15b000 0x7f574c0d0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574bfff700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f574c15b000 0x7f574c0d0b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53572/0 (socket says 172.21.15.67:53572) 2023-06-23T01:24:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574bfff700 1 -- 172.21.15.67:0/2400826653 learned_addr learned my addr 172.21.15.67:0/2400826653 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f575179d700 1 --2- 172.21.15.67:0/2400826653 >> v2:172.21.15.193:3300/0 conn(0x7f574c0d10b0 0x7f574c06cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.152 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574bfff700 1 -- 172.21.15.67:0/2400826653 >> v2:172.21.15.193:3300/0 conn(0x7f574c0d10b0 msgr2=0x7f574c06cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574bfff700 1 --2- 172.21.15.67:0/2400826653 >> v2:172.21.15.193:3300/0 conn(0x7f574c0d10b0 0x7f574c06cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574bfff700 1 -- 172.21.15.67:0/2400826653 >> v1:172.21.15.193:6789/0 conn(0x7f574c159c20 legacy=0x7f574c06e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574bfff700 1 -- 172.21.15.67:0/2400826653 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574c15d970 con 0x7f574c15b000 2023-06-23T01:24:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574bfff700 1 --2- 172.21.15.67:0/2400826653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c0d0b70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f574c15d8d0 tx=0x7f573c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19a2b47cab647f15 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574b7fe700 1 -- 172.21.15.67:0/2400826653 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c00e070 con 0x7f574c15b000 2023-06-23T01:24:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574b7fe700 1 -- 172.21.15.67:0/2400826653 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573c00ba20 con 0x7f574c15b000 2023-06-23T01:24:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.148+0000 7f574b7fe700 1 -- 172.21.15.67:0/2400826653 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573c0092e0 con 0x7f574c15b000 2023-06-23T01:24:13.153 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.149+0000 7f5751f9e700 1 -- 172.21.15.67:0/2400826653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 msgr2=0x7f574c0d0b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.149+0000 7f5751f9e700 1 --2- 172.21.15.67:0/2400826653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c0d0b70 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f574c15d8d0 tx=0x7f573c009840 comp rx=0 tx=0).stop 2023-06-23T01:24:13.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.149+0000 7f5751f9e700 1 -- 172.21.15.67:0/2400826653 shutdown_connections 2023-06-23T01:24:13.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.149+0000 7f5751f9e700 1 --2- 172.21.15.67:0/2400826653 >> v2:172.21.15.193:3300/0 conn(0x7f574c0d10b0 0x7f574c06cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.149+0000 7f5751f9e700 1 --2- 172.21.15.67:0/2400826653 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c0d0b70 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.149+0000 7f5751f9e700 1 -- 172.21.15.67:0/2400826653 >> 172.21.15.67:0/2400826653 conn(0x7f574c063020 msgr2=0x7f574c063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:13.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.149+0000 7f5751f9e700 1 -- 172.21.15.67:0/2400826653 shutdown_connections 2023-06-23T01:24:13.154 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.149+0000 7f5751f9e700 1 -- 172.21.15.67:0/2400826653 wait complete. 2023-06-23T01:24:13.156 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.150+0000 7f5751f9e700 1 Processor -- start 2023-06-23T01:24:13.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.150+0000 7f5751f9e700 1 -- start start 2023-06-23T01:24:13.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f5751f9e700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f574c0d10b0 0x7f574c158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f5751f9e700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f574c159c20 0x7f574c0c62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f5751f9e700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c15e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f5751f9e700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c0c6cd0 con 0x7f574c15b000 2023-06-23T01:24:13.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f5751f9e700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574c155c90 con 0x7f574c159c20 2023-06-23T01:24:13.157 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f5751f9e700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574c155f50 con 0x7f574c0d10b0 2023-06-23T01:24:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c15e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c15e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53580/0 (socket says 172.21.15.67:53580) 2023-06-23T01:24:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 -- 172.21.15.67:0/66803421 learned_addr learned my addr 172.21.15.67:0/66803421 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 -- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f574c0d10b0 msgr2=0x7f574c158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f574c0d10b0 0x7f574c158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 -- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f574c159c20 msgr2=0x7f574c0c62c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f574bfff700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f574c159c20 0x7f574c0c62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.158 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f574c159c20 0x7f574c0c62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 -- 172.21.15.67:0/66803421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c009ae0 con 0x7f574c15b000 2023-06-23T01:24:13.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.151+0000 7f575179d700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c15e9b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f573400ea50 tx=0x7f57340091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.152+0000 7f5748ff9700 1 -- 172.21.15.67:0/66803421 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573400c920 con 0x7f574c15b000 2023-06-23T01:24:13.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.152+0000 7f5748ff9700 1 -- 172.21.15.67:0/66803421 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f573400caa0 con 0x7f574c15b000 2023-06-23T01:24:13.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.152+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574c156230 con 0x7f574c15b000 2023-06-23T01:24:13.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.152+0000 7f5748ff9700 1 -- 172.21.15.67:0/66803421 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573400ce60 con 0x7f574c15b000 2023-06-23T01:24:13.159 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.152+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574c158ed0 con 0x7f574c15b000 2023-06-23T01:24:13.161 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.153+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574c05f060 con 0x7f574c15b000 2023-06-23T01:24:13.161 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.153+0000 7f5748ff9700 1 -- 172.21.15.67:0/66803421 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f5734013680 con 0x7f574c15b000 2023-06-23T01:24:13.161 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.154+0000 7f5748ff9700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f572c066e90 0x7f572c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.161 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.154+0000 7f5748ff9700 1 -- 172.21.15.67:0/66803421 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f5734084950 con 0x7f574c15b000 2023-06-23T01:24:13.161 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.154+0000 7f5750f9c700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f572c066e90 0x7f572c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.154+0000 7f5750f9c700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f572c066e90 0x7f572c069340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5744001730 tx=0x7f5744007b40 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.162 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.158+0000 7f5748ff9700 1 -- 172.21.15.67:0/66803421 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f5734051b80 con 0x7f574c15b000 2023-06-23T01:24:13.403 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.400+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f574c0d1850 con 0x7f574c15b000 2023-06-23T01:24:13.408 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.406+0000 7f5748ff9700 1 -- 172.21.15.67:0/66803421 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f57340551a0 con 0x7f574c15b000 2023-06-23T01:24:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.410+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f572c066e90 msgr2=0x7f572c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.410+0000 7f5751f9e700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f572c066e90 0x7f572c069340 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5744001730 tx=0x7f5744007b40 comp rx=0 tx=0).stop 2023-06-23T01:24:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.410+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 msgr2=0x7f574c15e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.410+0000 7f5751f9e700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c15e9b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f573400ea50 tx=0x7f57340091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.413 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.411+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 shutdown_connections 2023-06-23T01:24:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.411+0000 7f5751f9e700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f572c066e90 0x7f572c069340 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5744001730 tx=0x7f5744007b40 comp rx=0 tx=0).stop 2023-06-23T01:24:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.411+0000 7f5751f9e700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f574c15b000 0x7f574c15e9b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.411+0000 7f5751f9e700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f574c0d10b0 0x7f574c158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.411+0000 7f5751f9e700 1 --2- 172.21.15.67:0/66803421 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f574c159c20 0x7f574c0c62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.414 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.411+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 >> 172.21.15.67:0/66803421 conn(0x7f574c063020 msgr2=0x7f574c0b6680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:13.415 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.413+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 shutdown_connections 2023-06-23T01:24:13.416 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.414+0000 7f5751f9e700 1 -- 172.21.15.67:0/66803421 wait complete. 2023-06-23T01:24:13.422 INFO:teuthology.orchestra.run.smithi067.stderr:new fs with metadata pool 3 and data pool 4 2023-06-23T01:24:13.435 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:24:13.553 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.551+0000 7f4bd085d700 1 Processor -- start 2023-06-23T01:24:13.554 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.551+0000 7f4bd085d700 1 -- start start 2023-06-23T01:24:13.555 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.552+0000 7f4bd085d700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bcc159eb0 0x7f4bcc0cb1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.555 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.552+0000 7f4bd085d700 1 --2- >> v2:172.21.15.193:3300/0 conn(0x7f4bcc0cb6e0 0x7f4bcc15dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.556 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.552+0000 7f4bd085d700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bcc0cbb00 con 0x7f4bcc159ae0 2023-06-23T01:24:13.556 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.552+0000 7f4bd085d700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bcc15e8f0 con 0x7f4bcc159eb0 2023-06-23T01:24:13.556 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.552+0000 7f4bd085d700 1 -- --> v2:172.21.15.193:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bcc0c5c30 con 0x7f4bcc0cb6e0 2023-06-23T01:24:13.556 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.552+0000 7f4bca59c700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f4bcc159ae0 0x7f4bcc0c69f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60058/0 (socket says 172.21.15.67:60058) 2023-06-23T01:24:13.556 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bca59c700 1 -- 172.21.15.67:0/2944491769 learned_addr learned my addr 172.21.15.67:0/2944491769 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:13.556 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.552+0000 7f4bc9d9b700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f4bcc159eb0 0x7f4bcc0cb1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.556 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc959a700 1 -- 172.21.15.67:0/2944491769 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2924709501 0 0) 0x7f4bcc0cbb00 con 0x7f4bcc159ae0 2023-06-23T01:24:13.557 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc959a700 1 -- 172.21.15.67:0/2944491769 --> v1:172.21.15.67:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bb8003570 con 0x7f4bcc159ae0 2023-06-23T01:24:13.557 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bcad9d700 1 --2- 172.21.15.67:0/2944491769 >> v2:172.21.15.193:3300/0 conn(0x7f4bcc0cb6e0 0x7f4bcc15dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.557 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc9d9b700 1 -- 172.21.15.67:0/2944491769 >> v2:172.21.15.193:3300/0 conn(0x7f4bcc0cb6e0 msgr2=0x7f4bcc15dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:13.557 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc9d9b700 1 --2- 172.21.15.67:0/2944491769 >> v2:172.21.15.193:3300/0 conn(0x7f4bcc0cb6e0 0x7f4bcc15dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.557 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc9d9b700 1 -- 172.21.15.67:0/2944491769 >> v1:172.21.15.67:6789/0 conn(0x7f4bcc159ae0 legacy=0x7f4bcc0c69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.557 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc9d9b700 1 -- 172.21.15.67:0/2944491769 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bcc0c5f10 con 0x7f4bcc159eb0 2023-06-23T01:24:13.557 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc9d9b700 1 --2- 172.21.15.67:0/2944491769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc0cb1a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0ccc90 tx=0x7f4bb40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8ce1c40c434ee4 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc959a700 1 -- 172.21.15.67:0/2944491769 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb4011630 con 0x7f4bcc159eb0 2023-06-23T01:24:13.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc959a700 1 -- 172.21.15.67:0/2944491769 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb4011790 con 0x7f4bcc159eb0 2023-06-23T01:24:13.558 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.553+0000 7f4bc959a700 1 -- 172.21.15.67:0/2944491769 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb4011b30 con 0x7f4bcc159eb0 2023-06-23T01:24:13.560 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.554+0000 7f4bd085d700 1 -- 172.21.15.67:0/2944491769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 msgr2=0x7f4bcc0cb1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.560 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.554+0000 7f4bd085d700 1 --2- 172.21.15.67:0/2944491769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc0cb1a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc0ccc90 tx=0x7f4bb40091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.560 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.554+0000 7f4bd085d700 1 -- 172.21.15.67:0/2944491769 shutdown_connections 2023-06-23T01:24:13.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.554+0000 7f4bd085d700 1 --2- 172.21.15.67:0/2944491769 >> v2:172.21.15.193:3300/0 conn(0x7f4bcc0cb6e0 0x7f4bcc15dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.554+0000 7f4bd085d700 1 --2- 172.21.15.67:0/2944491769 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc0cb1a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.561 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.554+0000 7f4bd085d700 1 -- 172.21.15.67:0/2944491769 >> 172.21.15.67:0/2944491769 conn(0x7f4bcc063020 msgr2=0x7f4bcc063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.554+0000 7f4bd085d700 1 -- 172.21.15.67:0/2944491769 shutdown_connections 2023-06-23T01:24:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.554+0000 7f4bd085d700 1 -- 172.21.15.67:0/2944491769 wait complete. 2023-06-23T01:24:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.555+0000 7f4bd085d700 1 Processor -- start 2023-06-23T01:24:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.555+0000 7f4bd085d700 1 -- start start 2023-06-23T01:24:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bd085d700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bcc0cb6e0 0x7f4bcc152020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bd085d700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bcc159ae0 0x7f4bcc06c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bd085d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc06e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.562 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bd085d700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc1698f0 con 0x7f4bcc159eb0 2023-06-23T01:24:13.563 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bd085d700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc06d720 con 0x7f4bcc0cb6e0 2023-06-23T01:24:13.563 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bd085d700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bcc0b5fc0 con 0x7f4bcc159ae0 2023-06-23T01:24:13.563 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc06e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.563 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc06e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53598/0 (socket says 172.21.15.67:53598) 2023-06-23T01:24:13.563 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 -- 172.21.15.67:0/654247455 learned_addr learned my addr 172.21.15.67:0/654247455 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:13.563 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bca59c700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bcc0cb6e0 0x7f4bcc152020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.563 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bc9d9b700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bcc159ae0 0x7f4bcc06c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 -- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bcc159ae0 msgr2=0x7f4bcc06c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bcc159ae0 0x7f4bcc06c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 -- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bcc0cb6e0 msgr2=0x7f4bcc152020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bcc0cb6e0 0x7f4bcc152020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 -- 172.21.15.67:0/654247455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb4009420 con 0x7f4bcc159eb0 2023-06-23T01:24:13.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bc9d9b700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bcc159ae0 0x7f4bcc06c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:24:13.564 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bcad9d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc06e670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00c880 tx=0x7f4bbc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bc27fc700 1 -- 172.21.15.67:0/654247455 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc011600 con 0x7f4bcc159eb0 2023-06-23T01:24:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bc27fc700 1 -- 172.21.15.67:0/654247455 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bbc011760 con 0x7f4bcc159eb0 2023-06-23T01:24:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bc27fc700 1 -- 172.21.15.67:0/654247455 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bbc011ae0 con 0x7f4bcc159eb0 2023-06-23T01:24:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bcc06ebb0 con 0x7f4bcc159eb0 2023-06-23T01:24:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.556+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bcc06d160 con 0x7f4bcc159eb0 2023-06-23T01:24:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.557+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bcc05f060 con 0x7f4bcc159eb0 2023-06-23T01:24:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.558+0000 7f4bc27fc700 1 -- 172.21.15.67:0/654247455 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f4bbc0193f0 con 0x7f4bcc159eb0 2023-06-23T01:24:13.565 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.558+0000 7f4bc27fc700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f4bac066e90 0x7f4bac069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.558+0000 7f4bc27fc700 1 -- 172.21.15.67:0/654247455 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4bbc0847c0 con 0x7f4bcc159eb0 2023-06-23T01:24:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.558+0000 7f4bca59c700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f4bac066e90 0x7f4bac069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.566 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.559+0000 7f4bca59c700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f4bac066e90 0x7f4bac069340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc06e090 tx=0x7f4bc4009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.567 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.563+0000 7f4bc27fc700 1 -- 172.21.15.67:0/654247455 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f4bbc051990 con 0x7f4bcc159eb0 2023-06-23T01:24:13.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.758+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4bcc06ebb0 con 0x7f4bcc159eb0 2023-06-23T01:24:13.761 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.759+0000 7f4bc27fc700 1 -- 172.21.15.67:0/654247455 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16927 (secure 0 0 0) 0x7f4bcc06ebb0 con 0x7f4bcc159eb0 2023-06-23T01:24:13.762 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:24:13.762 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"0068cd93-d7dd-4502-a85d-c8368ea7e9d2","created":"2023-06-23T01:23:54.450197+0000","modified":"2023-06-23T01:24:13.401361+0000","last_up_change":"2023-06-23T01:24:00.795797+0000","last_in_change":"2023-06-23T01:23:57.782500+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.69999998807907104,"backfillfull_ratio":0.60000002384185791,"nearfull_ratio":0.60000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:24:01.045792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:24:06.629871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:24:11.033286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T01:24:12.367815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":111550},{"type":"v1","addr":"172.21.15.67:6801","nonce":111550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":111550},{"type":"v1","addr":"172.21.15.67:6803","nonce":111550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":111550},{"type":"v1","addr":"172.21.15.67:6807","nonce":111550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":111550},{"type":"v1","addr":"172.21.15.67:6805","nonce":111550}]},"public_addr":"172.21.15.67:6801/111550","cluster_addr":"172.21.15.67:6803/111550","heartbeat_back_addr":"172.21.15.67:6807/111550","heartbeat_front_addr":"172.21.15.67:6805/111550","state":["exists","up"]},{"osd":1,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":111554},{"type":"v1","addr":"172.21.15.67:6812","nonce":111554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":111554},{"type":"v1","addr":"172.21.15.67:6816","nonce":111554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":111554},{"type":"v1","addr":"172.21.15.67:6823","nonce":111554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":111554},{"type":"v1","addr":"172.21.15.67:6820","nonce":111554}]},"public_addr":"172.21.15.67:6812/111554","cluster_addr":"172.21.15.67:6816/111554","heartbeat_back_addr":"172.21.15.67:6823/111554","heartbeat_front_addr":"172.21.15.67:6820/111554","state":["exists","up"]},{"osd":2,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":111558},{"type":"v1","addr":"172.21.15.67:6809","nonce":111558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":111558},{"type":"v1","addr":"172.21.15.67:6813","nonce":111558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":111558},{"type":"v1","addr":"172.21.15.67:6821","nonce":111558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":111558},{"type":"v1","addr":"172.21.15.67:6817","nonce":111558}]},"public_addr":"172.21.15.67:6809/111558","cluster_addr":"172.21.15.67:6813/111558","heartbeat_back_addr":"172.21.15.67:6821/111558","heartbeat_front_addr":"172.21.15.67:6817/111558","state":["exists","up"]},{"osd":3,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":111567},{"type":"v1","addr":"172.21.15.67:6825","nonce":111567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":111567},{"type":"v1","addr":"172.21.15.67:6827","nonce":111567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":111567},{"type":"v1","addr":"172.21.15.67:6831","nonce":111567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":111567},{"type":"v1","addr":"172.21.15.67:6829","nonce":111567}]},"public_addr":"172.21.15.67:6825/111567","cluster_addr":"172.21.15.67:6827/111567","heartbeat_back_addr":"172.21.15.67:6831/111567","heartbeat_front_addr":"172.21.15.67:6829/111567","state":["exists","up"]},{"osd":4,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":83673},{"type":"v1","addr":"172.21.15.193:6801","nonce":83673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":83673},{"type":"v1","addr":"172.21.15.193:6803","nonce":83673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":83673},{"type":"v1","addr":"172.21.15.193:6807","nonce":83673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":83673},{"type":"v1","addr":"172.21.15.193:6805","nonce":83673}]},"public_addr":"172.21.15.193:6801/83673","cluster_addr":"172.21.15.193:6803/83673","heartbeat_back_addr":"172.21.15.193:6807/83673","heartbeat_front_addr":"172.21.15.193:6805/83673","state":["exists","up"]},{"osd":5,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83674},{"type":"v1","addr":"172.21.15.193:6809","nonce":83674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83674},{"type":"v1","addr":"172.21.15.193:6811","nonce":83674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83674},{"type":"v1","addr":"172.21.15.193:6815","nonce":83674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83674},{"type":"v1","addr":"172.21.15.193:6813","nonce":83674}]},"public_addr":"172.21.15.193:6809/83674","cluster_addr":"172.21.15.193:6811/83674","heartbeat_back_addr":"172.21.15.193:6815/83674","heartbeat_front_addr":"172.21.15.193:6813/83674","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":83699},{"type":"v1","addr":"172.21.15.193:6825","nonce":83699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":83699},{"type":"v1","addr":"172.21.15.193:6827","nonce":83699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":83699},{"type":"v1","addr":"172.21.15.193:6831","nonce":83699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":83699},{"type":"v1","addr":"172.21.15.193:6829","nonce":83699}]},"public_addr":"172.21.15.193:6825/83699","cluster_addr":"172.21.15.193:6827/83699","heartbeat_back_addr":"172.21.15.193:6831/83699","heartbeat_front_addr":"172.21.15.193:6829/83699","state":["exists","up"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":83675},{"type":"v1","addr":"172.21.15.193:6817","nonce":83675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":83675},{"type":"v1","addr":"172.21.15.193:6819","nonce":83675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":83675},{"type":"v1","addr":"172.21.15.193:6823","nonce":83675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":83675},{"type":"v1","addr":"172.21.15.193:6821","nonce":83675}]},"public_addr":"172.21.15.193:6817/83675","cluster_addr":"172.21.15.193:6819/83675","heartbeat_back_addr":"172.21.15.193:6823/83675","heartbeat_front_addr":"172.21.15.193:6821/83675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.993918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.906223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.928802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.986013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.876314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.922270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.983980+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.902644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:24:13.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.761+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f4bac066e90 msgr2=0x7f4bac069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.764 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.761+0000 7f4bd085d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f4bac066e90 0x7f4bac069340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc06e090 tx=0x7f4bc4009040 comp rx=0 tx=0).stop 2023-06-23T01:24:13.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.761+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 msgr2=0x7f4bcc06e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.761+0000 7f4bd085d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc06e670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4bbc00c880 tx=0x7f4bbc008040 comp rx=0 tx=0).stop 2023-06-23T01:24:13.765 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.762+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 shutdown_connections 2023-06-23T01:24:13.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.762+0000 7f4bd085d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f4bac066e90 0x7f4bac069340 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.762+0000 7f4bd085d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f4bcc159eb0 0x7f4bcc06e670 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.762+0000 7f4bd085d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4bcc0cb6e0 0x7f4bcc152020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.762+0000 7f4bd085d700 1 --2- 172.21.15.67:0/654247455 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f4bcc159ae0 0x7f4bcc06c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.762+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 >> 172.21.15.67:0/654247455 conn(0x7f4bcc063020 msgr2=0x7f4bcc158750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:13.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.762+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 shutdown_connections 2023-06-23T01:24:13.766 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.762+0000 7f4bd085d700 1 -- 172.21.15.67:0/654247455 wait complete. 2023-06-23T01:24:13.776 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:24:13.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.892+0000 7f34db776700 1 Processor -- start 2023-06-23T01:24:13.895 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.892+0000 7f34db776700 1 -- start start 2023-06-23T01:24:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.893+0000 7f34db776700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34d4157e30 0x7f34d40aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.893+0000 7f34db776700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40af650 con 0x7f34d40af1f0 2023-06-23T01:24:13.902 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.893+0000 7f34db776700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d40cab40 con 0x7f34d4157a60 2023-06-23T01:24:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.893+0000 7f34db776700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34d40b1130 con 0x7f34d4157e30 2023-06-23T01:24:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.893+0000 7f34d8d11700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f34d4157e30 0x7f34d40aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.893+0000 7f34d9d13700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f34d40af1f0 0x7f34d40cafd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60060/0 (socket says 172.21.15.67:60060) 2023-06-23T01:24:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.893+0000 7f34d9d13700 1 -- 172.21.15.67:0/1745345727 learned_addr learned my addr 172.21.15.67:0/1745345727 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34d8d11700 1 -- 172.21.15.67:0/1745345727 >> v1:172.21.15.193:6789/0 conn(0x7f34d4157a60 legacy=0x7f34d40afe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.903 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34d8d11700 1 -- 172.21.15.67:0/1745345727 >> v1:172.21.15.67:6789/0 conn(0x7f34d40af1f0 legacy=0x7f34d40cafd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34d8d11700 1 -- 172.21.15.67:0/1745345727 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d40ca810 con 0x7f34d4157e30 2023-06-23T01:24:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34d3fff700 1 -- 172.21.15.67:0/1745345727 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 213551919 0 0) 0x7f34c4001490 con 0x7f34d40af1f0 2023-06-23T01:24:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34d8d11700 1 --2- 172.21.15.67:0/1745345727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40aecb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f34c80098e0 tx=0x7f34c80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78542faddcbd546 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34d3fff700 1 -- 172.21.15.67:0/1745345727 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c8009e80 con 0x7f34d4157e30 2023-06-23T01:24:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34d3fff700 1 -- 172.21.15.67:0/1745345727 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c800b860 con 0x7f34d4157e30 2023-06-23T01:24:13.904 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34d3fff700 1 -- 172.21.15.67:0/1745345727 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c800bbe0 con 0x7f34d4157e30 2023-06-23T01:24:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34db776700 1 -- 172.21.15.67:0/1745345727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 msgr2=0x7f34d40aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34db776700 1 --2- 172.21.15.67:0/1745345727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40aecb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f34c80098e0 tx=0x7f34c80091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34db776700 1 -- 172.21.15.67:0/1745345727 shutdown_connections 2023-06-23T01:24:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34db776700 1 --2- 172.21.15.67:0/1745345727 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40aecb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.894+0000 7f34db776700 1 -- 172.21.15.67:0/1745345727 >> 172.21.15.67:0/1745345727 conn(0x7f34d40cdb10 msgr2=0x7f34d4067420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.895+0000 7f34db776700 1 -- 172.21.15.67:0/1745345727 shutdown_connections 2023-06-23T01:24:13.905 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.895+0000 7f34db776700 1 -- 172.21.15.67:0/1745345727 wait complete. 2023-06-23T01:24:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.896+0000 7f34db776700 1 Processor -- start 2023-06-23T01:24:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.896+0000 7f34db776700 1 -- start start 2023-06-23T01:24:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.896+0000 7f34db776700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d40af1f0 0x7f34d415d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.896+0000 7f34db776700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f34d4157a60 0x7f34d4067b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.896+0000 7f34db776700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40ade10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.896+0000 7f34db776700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40680c0 con 0x7f34d4157e30 2023-06-23T01:24:13.906 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.896+0000 7f34db776700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d40682b0 con 0x7f34d40af1f0 2023-06-23T01:24:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.896+0000 7f34db776700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4068590 con 0x7f34d4157a60 2023-06-23T01:24:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40ade10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40ade10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53628/0 (socket says 172.21.15.67:53628) 2023-06-23T01:24:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 -- 172.21.15.67:0/3893570379 learned_addr learned my addr 172.21.15.67:0/3893570379 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d8d11700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f34d4157a60 0x7f34d4067b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 -- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f34d4157a60 msgr2=0x7f34d4067b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.907 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9512700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d40af1f0 0x7f34d415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f34d4157a60 0x7f34d4067b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 -- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d40af1f0 msgr2=0x7f34d415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d40af1f0 0x7f34d415d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 -- 172.21.15.67:0/3893570379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8009420 con 0x7f34d4157e30 2023-06-23T01:24:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9512700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d40af1f0 0x7f34d415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:24:13.908 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d9d13700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40ade10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f34c400cbf0 tx=0x7f34c40019b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d17fa700 1 -- 172.21.15.67:0/3893570379 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c400a9c0 con 0x7f34d4157e30 2023-06-23T01:24:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d17fa700 1 -- 172.21.15.67:0/3893570379 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34c400ab20 con 0x7f34d4157e30 2023-06-23T01:24:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4068870 con 0x7f34d4157e30 2023-06-23T01:24:13.909 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34d17fa700 1 -- 172.21.15.67:0/3893570379 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34c40070f0 con 0x7f34d4157e30 2023-06-23T01:24:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.897+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d4068e30 con 0x7f34d4157e30 2023-06-23T01:24:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.898+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d40691a0 con 0x7f34d4157e30 2023-06-23T01:24:13.910 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.899+0000 7f34d17fa700 1 -- 172.21.15.67:0/3893570379 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f34c4007520 con 0x7f34d4157e30 2023-06-23T01:24:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.899+0000 7f34d17fa700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f34c0066e90 0x7f34c0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.899+0000 7f34d17fa700 1 -- 172.21.15.67:0/3893570379 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f34c40847b0 con 0x7f34d4157e30 2023-06-23T01:24:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.899+0000 7f34d9512700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f34c0066e90 0x7f34c0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:13.911 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.900+0000 7f34d9512700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f34c0066e90 0x7f34c0069340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f34cc001840 tx=0x7f34cc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:13.912 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:13.903+0000 7f34d17fa700 1 -- 172.21.15.67:0/3893570379 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f34c4051a00 con 0x7f34d4157e30 2023-06-23T01:24:14.105 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.103+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34d40af650 con 0x7f34d4157e30 2023-06-23T01:24:14.106 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.104+0000 7f34d17fa700 1 -- 172.21.15.67:0/3893570379 <== mon.0 v2:172.21.15.67:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16927 (secure 0 0 0) 0x7f34d40af650 con 0x7f34d4157e30 2023-06-23T01:24:14.107 INFO:teuthology.orchestra.run.smithi067.stdout: 2023-06-23T01:24:14.107 INFO:teuthology.orchestra.run.smithi067.stdout:{"epoch":22,"fsid":"0068cd93-d7dd-4502-a85d-c8368ea7e9d2","created":"2023-06-23T01:23:54.450197+0000","modified":"2023-06-23T01:24:13.401361+0000","last_up_change":"2023-06-23T01:24:00.795797+0000","last_in_change":"2023-06-23T01:23:57.782500+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.69999998807907104,"backfillfull_ratio":0.60000002384185791,"nearfull_ratio":0.60000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:24:01.045792+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:24:06.629871+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:24:11.033286+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T01:24:12.367815+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6800","nonce":111550},{"type":"v1","addr":"172.21.15.67:6801","nonce":111550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6802","nonce":111550},{"type":"v1","addr":"172.21.15.67:6803","nonce":111550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6806","nonce":111550},{"type":"v1","addr":"172.21.15.67:6807","nonce":111550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6804","nonce":111550},{"type":"v1","addr":"172.21.15.67:6805","nonce":111550}]},"public_addr":"172.21.15.67:6801/111550","cluster_addr":"172.21.15.67:6803/111550","heartbeat_back_addr":"172.21.15.67:6807/111550","heartbeat_front_addr":"172.21.15.67:6805/111550","state":["exists","up"]},{"osd":1,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6810","nonce":111554},{"type":"v1","addr":"172.21.15.67:6812","nonce":111554}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6814","nonce":111554},{"type":"v1","addr":"172.21.15.67:6816","nonce":111554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6822","nonce":111554},{"type":"v1","addr":"172.21.15.67:6823","nonce":111554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6818","nonce":111554},{"type":"v1","addr":"172.21.15.67:6820","nonce":111554}]},"public_addr":"172.21.15.67:6812/111554","cluster_addr":"172.21.15.67:6816/111554","heartbeat_back_addr":"172.21.15.67:6823/111554","heartbeat_front_addr":"172.21.15.67:6820/111554","state":["exists","up"]},{"osd":2,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6808","nonce":111558},{"type":"v1","addr":"172.21.15.67:6809","nonce":111558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6811","nonce":111558},{"type":"v1","addr":"172.21.15.67:6813","nonce":111558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6819","nonce":111558},{"type":"v1","addr":"172.21.15.67:6821","nonce":111558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6815","nonce":111558},{"type":"v1","addr":"172.21.15.67:6817","nonce":111558}]},"public_addr":"172.21.15.67:6809/111558","cluster_addr":"172.21.15.67:6813/111558","heartbeat_back_addr":"172.21.15.67:6821/111558","heartbeat_front_addr":"172.21.15.67:6817/111558","state":["exists","up"]},{"osd":3,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6824","nonce":111567},{"type":"v1","addr":"172.21.15.67:6825","nonce":111567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6826","nonce":111567},{"type":"v1","addr":"172.21.15.67:6827","nonce":111567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6830","nonce":111567},{"type":"v1","addr":"172.21.15.67:6831","nonce":111567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.67:6828","nonce":111567},{"type":"v1","addr":"172.21.15.67:6829","nonce":111567}]},"public_addr":"172.21.15.67:6825/111567","cluster_addr":"172.21.15.67:6827/111567","heartbeat_back_addr":"172.21.15.67:6831/111567","heartbeat_front_addr":"172.21.15.67:6829/111567","state":["exists","up"]},{"osd":4,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":83673},{"type":"v1","addr":"172.21.15.193:6801","nonce":83673}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":83673},{"type":"v1","addr":"172.21.15.193:6803","nonce":83673}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":83673},{"type":"v1","addr":"172.21.15.193:6807","nonce":83673}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":83673},{"type":"v1","addr":"172.21.15.193:6805","nonce":83673}]},"public_addr":"172.21.15.193:6801/83673","cluster_addr":"172.21.15.193:6803/83673","heartbeat_back_addr":"172.21.15.193:6807/83673","heartbeat_front_addr":"172.21.15.193:6805/83673","state":["exists","up"]},{"osd":5,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":83674},{"type":"v1","addr":"172.21.15.193:6809","nonce":83674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":83674},{"type":"v1","addr":"172.21.15.193:6811","nonce":83674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":83674},{"type":"v1","addr":"172.21.15.193:6815","nonce":83674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":83674},{"type":"v1","addr":"172.21.15.193:6813","nonce":83674}]},"public_addr":"172.21.15.193:6809/83674","cluster_addr":"172.21.15.193:6811/83674","heartbeat_back_addr":"172.21.15.193:6815/83674","heartbeat_front_addr":"172.21.15.193:6813/83674","state":["exists","up"]},{"osd":6,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":83699},{"type":"v1","addr":"172.21.15.193:6825","nonce":83699}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":83699},{"type":"v1","addr":"172.21.15.193:6827","nonce":83699}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":83699},{"type":"v1","addr":"172.21.15.193:6831","nonce":83699}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":83699},{"type":"v1","addr":"172.21.15.193:6829","nonce":83699}]},"public_addr":"172.21.15.193:6825/83699","cluster_addr":"172.21.15.193:6827/83699","heartbeat_back_addr":"172.21.15.193:6831/83699","heartbeat_front_addr":"172.21.15.193:6829/83699","state":["exists","up"]},{"osd":7,"uuid":"00000000-0000-0000-0000-000000000000","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":83675},{"type":"v1","addr":"172.21.15.193:6817","nonce":83675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":83675},{"type":"v1","addr":"172.21.15.193:6819","nonce":83675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":83675},{"type":"v1","addr":"172.21.15.193:6823","nonce":83675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":83675},{"type":"v1","addr":"172.21.15.193:6821","nonce":83675}]},"public_addr":"172.21.15.193:6817/83675","cluster_addr":"172.21.15.193:6819/83675","heartbeat_back_addr":"172.21.15.193:6823/83675","heartbeat_front_addr":"172.21.15.193:6821/83675","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.993918+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.906223+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.928802+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.986013+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.876314+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.922270+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.983980+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:23:58.902644+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:24:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f34c0066e90 msgr2=0x7f34c0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f34c0066e90 0x7f34c0069340 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f34cc001840 tx=0x7f34cc009040 comp rx=0 tx=0).stop 2023-06-23T01:24:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 msgr2=0x7f34d40ade10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40ade10 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f34c400cbf0 tx=0x7f34c40019b0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 shutdown_connections 2023-06-23T01:24:14.109 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f34c0066e90 0x7f34c0069340 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f34d4157e30 0x7f34d40ade10 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34d40af1f0 0x7f34d415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 --2- 172.21.15.67:0/3893570379 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f34d4157a60 0x7f34d4067b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.106+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 >> 172.21.15.67:0/3893570379 conn(0x7f34d40cdb10 msgr2=0x7f34d40cc180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.107+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 shutdown_connections 2023-06-23T01:24:14.110 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.107+0000 7f34db776700 1 -- 172.21.15.67:0/3893570379 wait complete. 2023-06-23T01:24:14.121 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-23T01:24:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.242+0000 7f560f478700 1 Processor -- start 2023-06-23T01:24:14.244 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.242+0000 7f560f478700 1 -- start start 2023-06-23T01:24:14.246 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560f478700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5608159c20 0x7f560806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560f478700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560815e820 con 0x7f560806dd50 2023-06-23T01:24:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560f478700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f560815d970 con 0x7f56080d0b70 2023-06-23T01:24:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560f478700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56080b55e0 con 0x7f5608159c20 2023-06-23T01:24:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560e476700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7f5608159c20 0x7f560806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560ec77700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7f560806dd50 0x7f560815a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60064/0 (socket says 172.21.15.67:60064) 2023-06-23T01:24:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560ec77700 1 -- 172.21.15.67:0/2105676892 learned_addr learned my addr 172.21.15.67:0/2105676892 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:14.247 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560e476700 1 -- 172.21.15.67:0/2105676892 >> v1:172.21.15.193:6789/0 conn(0x7f56080d0b70 legacy=0x7f5608159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:24:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560e476700 1 -- 172.21.15.67:0/2105676892 >> v1:172.21.15.67:6789/0 conn(0x7f560806dd50 legacy=0x7f560815a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560d474700 1 -- 172.21.15.67:0/2105676892 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2840280562 0 0) 0x7f560815e820 con 0x7f560806dd50 2023-06-23T01:24:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.243+0000 7f560e476700 1 -- 172.21.15.67:0/2105676892 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560815dc10 con 0x7f5608159c20 2023-06-23T01:24:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560e476700 1 --2- 172.21.15.67:0/2105676892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5608159c20 0x7f560806e9e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5600009e60 tx=0x7f5600009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51e984212c1e6f28 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560d474700 1 -- 172.21.15.67:0/2105676892 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560000e070 con 0x7f5608159c20 2023-06-23T01:24:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560d474700 1 -- 172.21.15.67:0/2105676892 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560000bc30 con 0x7f5608159c20 2023-06-23T01:24:14.248 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560d474700 1 -- 172.21.15.67:0/2105676892 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5600012430 con 0x7f5608159c20 2023-06-23T01:24:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560f478700 1 -- 172.21.15.67:0/2105676892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5608159c20 msgr2=0x7f560806e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560f478700 1 --2- 172.21.15.67:0/2105676892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5608159c20 0x7f560806e9e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5600009e60 tx=0x7f5600009780 comp rx=0 tx=0).stop 2023-06-23T01:24:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560f478700 1 -- 172.21.15.67:0/2105676892 shutdown_connections 2023-06-23T01:24:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560f478700 1 --2- 172.21.15.67:0/2105676892 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f5608159c20 0x7f560806e9e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560f478700 1 -- 172.21.15.67:0/2105676892 >> 172.21.15.67:0/2105676892 conn(0x7f5608063020 msgr2=0x7f5608063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560f478700 1 -- 172.21.15.67:0/2105676892 shutdown_connections 2023-06-23T01:24:14.249 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.244+0000 7f560f478700 1 -- 172.21.15.67:0/2105676892 wait complete. 2023-06-23T01:24:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.245+0000 7f560f478700 1 Processor -- start 2023-06-23T01:24:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.245+0000 7f560f478700 1 -- start start 2023-06-23T01:24:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560f478700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560806dd50 0x7f56080c7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560f478700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56080d0b70 0x7f560816e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560f478700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5608159c20 0x7f5608144570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.252 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560f478700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5608158a00 con 0x7f560806dd50 2023-06-23T01:24:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560f478700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56080c6ba0 con 0x7f5608159c20 2023-06-23T01:24:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560f478700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f560816e840 con 0x7f56080d0b70 2023-06-23T01:24:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560e476700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560806dd50 0x7f56080c7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560e476700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560806dd50 0x7f56080c7030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53642/0 (socket says 172.21.15.67:53642) 2023-06-23T01:24:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560e476700 1 -- 172.21.15.67:0/4095435020 learned_addr learned my addr 172.21.15.67:0/4095435020 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:14.253 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560ec77700 1 --2- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5608159c20 0x7f5608144570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560dc75700 1 --2- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56080d0b70 0x7f560816e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560e476700 1 -- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56080d0b70 msgr2=0x7f560816e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560e476700 1 --2- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7f56080d0b70 0x7f560816e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560e476700 1 -- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5608159c20 msgr2=0x7f5608144570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560e476700 1 --2- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5608159c20 0x7f5608144570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.246+0000 7f560e476700 1 -- 172.21.15.67:0/4095435020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5600009a20 con 0x7f560806dd50 2023-06-23T01:24:14.254 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.247+0000 7f560e476700 1 --2- 172.21.15.67:0/4095435020 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7f560806dd50 0x7f56080c7030 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f56000099f0 tx=0x7f5600019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.247+0000 7f56067fc700 1 -- 172.21.15.67:0/4095435020 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560001b420 con 0x7f560806dd50 2023-06-23T01:24:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.247+0000 7f56067fc700 1 -- 172.21.15.67:0/4095435020 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560001b580 con 0x7f560806dd50 2023-06-23T01:24:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.247+0000 7f560f478700 1 -- 172.21.15.67:0/4095435020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56080c5f90 con 0x7f560806dd50 2023-06-23T01:24:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.247+0000 7f56067fc700 1 -- 172.21.15.67:0/4095435020 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f560001b940 con 0x7f560806dd50 2023-06-23T01:24:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.247+0000 7f560f478700 1 -- 172.21.15.67:0/4095435020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56080c6550 con 0x7f560806dd50 2023-06-23T01:24:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.248+0000 7f560f478700 1 -- 172.21.15.67:0/4095435020 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56080c6e80 con 0x7f560806dd50 2023-06-23T01:24:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.248+0000 7f56067fc700 1 -- 172.21.15.67:0/4095435020 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f560001bcf0 con 0x7f560806dd50 2023-06-23T01:24:14.255 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.248+0000 7f56067fc700 1 --2- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f55f4066e90 0x7f55f4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.248+0000 7f56067fc700 1 -- 172.21.15.67:0/4095435020 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f5600086350 con 0x7f560806dd50 2023-06-23T01:24:14.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.249+0000 7f560dc75700 1 --2- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f55f4066e90 0x7f55f4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.256 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.249+0000 7f560dc75700 1 --2- 172.21.15.67:0/4095435020 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7f55f4066e90 0x7f55f4069340 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f560806f150 tx=0x7f55fc009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:14.257 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.253+0000 7f56067fc700 1 -- 172.21.15.67:0/4095435020 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f5600053520 con 0x7f560806dd50 2023-06-23T01:24:14.498 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 1 closest matches: 2023-06-23T01:24:14.499 INFO:teuthology.orchestra.run.smithi067.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:24:14.499 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2023-06-23T01:24:14.501 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:24:14.502 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['disabled']} 2023-06-23T01:24:14.502 DEBUG:teuthology.orchestra.run.smithi067:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-23T01:24:14.628 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.623+0000 7ff5fdff8700 1 Processor -- start 2023-06-23T01:24:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.623+0000 7ff5fdff8700 1 -- start start 2023-06-23T01:24:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fdff8700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5f8159c20 0x7ff5f806e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fdff8700 1 -- --> v1:172.21.15.67:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f815e820 con 0x7ff5f806dd50 2023-06-23T01:24:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fdff8700 1 -- --> v1:172.21.15.193:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5f815d970 con 0x7ff5f80d0b70 2023-06-23T01:24:14.629 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fdff8700 1 -- --> v2:172.21.15.67:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5f80b55e0 con 0x7ff5f8159c20 2023-06-23T01:24:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fcff6700 1 --2- >> v2:172.21.15.67:3300/0 conn(0x7ff5f8159c20 0x7ff5f806e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fd7f7700 1 --1- >> v1:172.21.15.67:6789/0 conn(0x7ff5f806dd50 0x7ff5f815a700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.67:6789/0 says I am v1:172.21.15.67:60068/0 (socket says 172.21.15.67:60068) 2023-06-23T01:24:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fd7f7700 1 -- 172.21.15.67:0/1022263680 learned_addr learned my addr 172.21.15.67:0/1022263680 (peer_addr_for_me v1:172.21.15.67:0/0) 2023-06-23T01:24:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fcff6700 1 -- 172.21.15.67:0/1022263680 >> v1:172.21.15.193:6789/0 conn(0x7ff5f80d0b70 legacy=0x7ff5f8159ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fcff6700 1 -- 172.21.15.67:0/1022263680 >> v1:172.21.15.67:6789/0 conn(0x7ff5f806dd50 legacy=0x7ff5f815a700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fcff6700 1 -- 172.21.15.67:0/1022263680 --> v2:172.21.15.67:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5f815dc10 con 0x7ff5f8159c20 2023-06-23T01:24:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5f77fe700 1 -- 172.21.15.67:0/1022263680 <== mon.0 v1:172.21.15.67:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762379811 0 0) 0x7ff5f815e820 con 0x7ff5f806dd50 2023-06-23T01:24:14.630 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.624+0000 7ff5fcff6700 1 --2- 172.21.15.67:0/1022263680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f8159c20 0x7ff5f806e9e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80098e0 tx=0x7ff5e80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2f62f5655660954 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:14.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5f77fe700 1 -- 172.21.15.67:0/1022263680 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e8009e80 con 0x7ff5f8159c20 2023-06-23T01:24:14.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5f77fe700 1 -- 172.21.15.67:0/1022263680 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5e800b860 con 0x7ff5f8159c20 2023-06-23T01:24:14.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5f77fe700 1 -- 172.21.15.67:0/1022263680 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5e800bbe0 con 0x7ff5f8159c20 2023-06-23T01:24:14.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5fdff8700 1 -- 172.21.15.67:0/1022263680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f8159c20 msgr2=0x7ff5f806e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5fdff8700 1 --2- 172.21.15.67:0/1022263680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f8159c20 0x7ff5f806e9e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80098e0 tx=0x7ff5e80091a0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5fdff8700 1 -- 172.21.15.67:0/1022263680 shutdown_connections 2023-06-23T01:24:14.631 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5fdff8700 1 --2- 172.21.15.67:0/1022263680 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f8159c20 0x7ff5f806e9e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5fdff8700 1 -- 172.21.15.67:0/1022263680 >> 172.21.15.67:0/1022263680 conn(0x7ff5f8063020 msgr2=0x7ff5f8063420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:24:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5fdff8700 1 -- 172.21.15.67:0/1022263680 shutdown_connections 2023-06-23T01:24:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.625+0000 7ff5fdff8700 1 -- 172.21.15.67:0/1022263680 wait complete. 2023-06-23T01:24:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.626+0000 7ff5fdff8700 1 Processor -- start 2023-06-23T01:24:14.632 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5fdff8700 1 -- start start 2023-06-23T01:24:14.634 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5fdff8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5f806dd50 0x7ff5f8150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5fdff8700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f80d0b70 0x7ff5f813b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5fdff8700 1 --2- >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5f8159c20 0x7ff5f814f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5fdff8700 1 -- --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f814fed0 con 0x7ff5f80d0b70 2023-06-23T01:24:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5fdff8700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f81501b0 con 0x7ff5f806dd50 2023-06-23T01:24:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5fdff8700 1 -- --> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5f8150490 con 0x7ff5f8159c20 2023-06-23T01:24:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5f7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f80d0b70 0x7ff5f813b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.635 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5f7fff700 1 --2- >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f80d0b70 0x7ff5f813b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.67:3300/0 says I am v2:172.21.15.67:53666/0 (socket says 172.21.15.67:53666) 2023-06-23T01:24:14.637 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.627+0000 7ff5f7fff700 1 -- 172.21.15.67:0/1157654804 learned_addr learned my addr 172.21.15.67:0/1157654804 (peer_addr_for_me v2:172.21.15.67:0/0) 2023-06-23T01:24:14.637 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5fcff6700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5f806dd50 0x7ff5f8150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5fd7f7700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5f8159c20 0x7ff5f814f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f7fff700 1 -- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5f8159c20 msgr2=0x7ff5f814f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f7fff700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:3301/0,v1:172.21.15.193:6790/0] conn(0x7ff5f8159c20 0x7ff5f814f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f7fff700 1 -- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5f806dd50 msgr2=0x7ff5f8150f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:24:14.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f7fff700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5f806dd50 0x7ff5f8150f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:24:14.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f7fff700 1 -- 172.21.15.67:0/1157654804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e8009420 con 0x7ff5f80d0b70 2023-06-23T01:24:14.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5fcff6700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff5f806dd50 0x7ff5f8150f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:24:14.638 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f7fff700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] conn(0x7ff5f80d0b70 0x7ff5f813b820 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00c590 tx=0x7ff5ec00c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:14.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f4ff9700 1 -- 172.21.15.67:0/1157654804 <== mon.0 v2:172.21.15.67:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011590 con 0x7ff5f80d0b70 2023-06-23T01:24:14.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f4ff9700 1 -- 172.21.15.67:0/1157654804 <== mon.0 v2:172.21.15.67:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5ec0116f0 con 0x7ff5f80d0b70 2023-06-23T01:24:14.639 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5fdff8700 1 -- 172.21.15.67:0/1157654804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5f8150770 con 0x7ff5f80d0b70 2023-06-23T01:24:14.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5f4ff9700 1 -- 172.21.15.67:0/1157654804 <== mon.0 v2:172.21.15.67:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec011ab0 con 0x7ff5f80d0b70 2023-06-23T01:24:14.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.628+0000 7ff5fdff8700 1 -- 172.21.15.67:0/1157654804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5f813bd60 con 0x7ff5f80d0b70 2023-06-23T01:24:14.640 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.629+0000 7ff5fdff8700 1 -- 172.21.15.67:0/1157654804 --> [v2:172.21.15.67:3300/0,v1:172.21.15.67:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5f8150c10 con 0x7ff5f80d0b70 2023-06-23T01:24:14.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.629+0000 7ff5f4ff9700 1 -- 172.21.15.67:0/1157654804 <== mon.0 v2:172.21.15.67:3300/0 4 ==== mgrmap(e 4) v1 ==== 84648+0+0 (secure 0 0 0) 0x7ff5ec01b3b0 con 0x7ff5f80d0b70 2023-06-23T01:24:14.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.630+0000 7ff5f4ff9700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7ff5e4066e90 0x7ff5e4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:24:14.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.630+0000 7ff5f4ff9700 1 -- 172.21.15.67:0/1157654804 <== mon.0 v2:172.21.15.67:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff5ec083bb0 con 0x7ff5f80d0b70 2023-06-23T01:24:14.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.630+0000 7ff5fcff6700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7ff5e4066e90 0x7ff5e4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:24:14.641 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.630+0000 7ff5fcff6700 1 --2- 172.21.15.67:0/1157654804 >> [v2:172.21.15.193:6832/83088,v1:172.21.15.193:6833/83088] conn(0x7ff5e4066e90 0x7ff5e4069340 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff5e80098e0 tx=0x7ff5e8017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:24:14.642 INFO:teuthology.orchestra.run.smithi067.stderr:2023-06-23T01:24:14.634+0000 7ff5f4ff9700 1 -- 172.21.15.67:0/1157654804 <== mon.0 v2:172.21.15.67:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7ff5ec050d80 con 0x7ff5f80d0b70 2023-06-23T01:24:14.879 INFO:teuthology.orchestra.run.smithi067.stderr:no valid command found; 1 closest matches: 2023-06-23T01:24:14.879 INFO:teuthology.orchestra.run.smithi067.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:24:14.879 INFO:teuthology.orchestra.run.smithi067.stderr:Error EINVAL: invalid command 2023-06-23T01:24:14.882 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:24:14.882 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:24:14.883 INFO:teuthology.misc:Shutting down mds daemons... 2023-06-23T01:24:14.883 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-23T01:24:14.884 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:14.891 INFO:tasks.ceph.mds.a:Stopped 2023-06-23T01:24:14.892 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-23T01:24:14.892 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:14.940 INFO:tasks.ceph.mds.c:Stopped 2023-06-23T01:24:14.941 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-23T01:24:14.941 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:14.951 INFO:tasks.ceph.mds.b:Stopped 2023-06-23T01:24:14.952 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-23T01:24:14.952 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.004 INFO:tasks.ceph.mds.d:Stopped 2023-06-23T01:24:15.004 INFO:teuthology.misc:Shutting down osd daemons... 2023-06-23T01:24:15.005 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-06-23T01:24:15.005 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.015 INFO:tasks.ceph.osd.0:Stopped 2023-06-23T01:24:15.016 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-06-23T01:24:15.016 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.065 INFO:tasks.ceph.osd.1:Stopped 2023-06-23T01:24:15.066 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-06-23T01:24:15.066 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.116 INFO:tasks.ceph.osd.2:Stopped 2023-06-23T01:24:15.117 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-06-23T01:24:15.117 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.167 INFO:tasks.ceph.osd.3:Stopped 2023-06-23T01:24:15.168 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-06-23T01:24:15.168 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.180 INFO:tasks.ceph.osd.4:Stopped 2023-06-23T01:24:15.180 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-06-23T01:24:15.181 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.233 INFO:tasks.ceph.osd.5:Stopped 2023-06-23T01:24:15.234 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-06-23T01:24:15.234 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.285 INFO:tasks.ceph.osd.6:Stopped 2023-06-23T01:24:15.286 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-06-23T01:24:15.287 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.337 INFO:tasks.ceph.osd.7:Stopped 2023-06-23T01:24:15.339 INFO:teuthology.misc:Shutting down mgr daemons... 2023-06-23T01:24:15.339 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-06-23T01:24:15.340 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.363 INFO:tasks.ceph.mgr.y:Stopped 2023-06-23T01:24:15.363 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-06-23T01:24:15.364 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.402 INFO:tasks.ceph.mgr.x:Stopped 2023-06-23T01:24:15.403 INFO:teuthology.misc:Shutting down mon daemons... 2023-06-23T01:24:15.403 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-06-23T01:24:15.404 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.413 INFO:tasks.ceph.mon.a:Stopped 2023-06-23T01:24:15.413 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-06-23T01:24:15.414 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.452 INFO:tasks.ceph.mon.b:Stopped 2023-06-23T01:24:15.453 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-06-23T01:24:15.453 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:24:15.502 INFO:tasks.ceph.mon.c:Stopped 2023-06-23T01:24:15.503 INFO:tasks.ceph:Checking cluster log for badness... 2023-06-23T01:24:15.503 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | egrep -v PG_RECOVERY_FULL | egrep -v PG_DEGRADED | head -n 1 2023-06-23T01:24:15.536 INFO:teuthology.orchestra.run.smithi067.stdout:2023-06-23T01:23:59.807027+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 214026.528160 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. 2023-06-23T01:24:15.536 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-06-23T01:24:15.537 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | egrep -v PG_RECOVERY_FULL | egrep -v PG_DEGRADED | head -n 1 2023-06-23T01:24:15.607 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | egrep -v PG_RECOVERY_FULL | egrep -v PG_DEGRADED | head -n 1 2023-06-23T01:24:15.677 DEBUG:teuthology.orchestra.run.smithi067:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | egrep -v PG_RECOVERY_FULL | egrep -v PG_DEGRADED | head -n 1 2023-06-23T01:24:15.749 INFO:teuthology.orchestra.run.smithi067.stdout:2023-06-23T01:23:59.807027+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 214026.528160 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd's IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]. 2023-06-23T01:24:15.750 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:24:15.750 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-06-23T01:24:16.135 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:24:16.136 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-06-23T01:24:16.332 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:24:16.333 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-06-23T01:24:16.393 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:24:16.394 DEBUG:teuthology.orchestra.run.smithi067:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-06-23T01:24:16.485 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:24:16.485 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-06-23T01:24:17.089 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:24:17.089 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-06-23T01:24:17.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:24:17.158 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-06-23T01:24:17.224 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:24:17.225 DEBUG:teuthology.orchestra.run.smithi193:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-06-23T01:24:17.273 INFO:tasks.ceph:Archiving mon data... 2023-06-23T01:24:17.275 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/data/mon.a.tgz 2023-06-23T01:24:17.276 DEBUG:teuthology.orchestra.run.smithi067:> mktemp 2023-06-23T01:24:17.293 INFO:teuthology.orchestra.run.smithi067.stdout:/tmp/tmp.mpeetTc672 2023-06-23T01:24:17.294 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.mpeetTc672 2023-06-23T01:24:17.415 DEBUG:teuthology.orchestra.run.smithi067:> sudo chmod 0666 /tmp/tmp.mpeetTc672 2023-06-23T01:24:17.499 DEBUG:teuthology.orchestra.remote:smithi067:/tmp/tmp.mpeetTc672 is 162KB 2023-06-23T01:24:17.532 DEBUG:teuthology.orchestra.run.smithi067:> rm -fr /tmp/tmp.mpeetTc672 2023-06-23T01:24:17.547 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/data/mon.b.tgz 2023-06-23T01:24:17.547 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2023-06-23T01:24:17.563 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.DqbtK3Npwr 2023-06-23T01:24:17.563 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.DqbtK3Npwr 2023-06-23T01:24:17.683 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.DqbtK3Npwr 2023-06-23T01:24:17.764 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.DqbtK3Npwr is 225KB 2023-06-23T01:24:17.855 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.DqbtK3Npwr 2023-06-23T01:24:17.870 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/data/mon.c.tgz 2023-06-23T01:24:17.870 DEBUG:teuthology.orchestra.run.smithi193:> mktemp 2023-06-23T01:24:17.923 INFO:teuthology.orchestra.run.smithi193.stdout:/tmp/tmp.6GXPsb7Uk3 2023-06-23T01:24:17.923 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.6GXPsb7Uk3 2023-06-23T01:24:18.023 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 0666 /tmp/tmp.6GXPsb7Uk3 2023-06-23T01:24:18.105 DEBUG:teuthology.orchestra.remote:smithi193:/tmp/tmp.6GXPsb7Uk3 is 225KB 2023-06-23T01:24:18.171 DEBUG:teuthology.orchestra.run.smithi193:> rm -fr /tmp/tmp.6GXPsb7Uk3 2023-06-23T01:24:18.186 INFO:tasks.ceph:Cleaning ceph cluster... 2023-06-23T01:24:18.186 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T01:24:18.190 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T01:24:18.250 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-06-23T01:24:18.251 DEBUG:teuthology.orchestra.run.smithi067:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T01:24:18.258 DEBUG:teuthology.orchestra.run.smithi193:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T01:24:18.282 INFO:teuthology.orchestra.run.smithi067.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T01:24:18.315 INFO:teuthology.orchestra.run.smithi193.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T01:24:18.321 INFO:tasks.ceph:Archiving crash dumps... 2023-06-23T01:24:18.323 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/remote/smithi067/crash 2023-06-23T01:24:18.325 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T01:24:18.355 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/remote/smithi193/crash 2023-06-23T01:24:18.357 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T01:24:18.384 INFO:tasks.ceph:Shutting down logrotate 2023-06-23T01:24:18.385 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:24:18.400 DEBUG:teuthology.orchestra.run.smithi193:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:24:18.478 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T01:24:18.504 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T01:24:18.532 INFO:tasks.ceph:Compressing logs... 2023-06-23T01:24:18.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:24:18.547 DEBUG:teuthology.orchestra.run.smithi193:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:24:18.764 INFO:tasks.ceph:Archiving logs... 2023-06-23T01:24:18.764 DEBUG:teuthology.misc:Transferring archived files from smithi067:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/remote/smithi067/log 2023-06-23T01:24:18.765 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:24:18.824 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/remote/smithi193/log 2023-06-23T01:24:18.825 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:24:18.876 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:24:18.949 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=e7c0ca9bff0c40e4aa3fc630bd7b9b5e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi067 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:24:18.955 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-06-23T01:24:18.974 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-23T01:24:19.005 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-23T01:24:19.006 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:24:19.018 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:24:19.100 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-23T01:24:19.113 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-23T01:24:19.114 DEBUG:teuthology.orchestra.run.smithi067:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:24:19.116 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:24:19.128 INFO:teuthology.orchestra.run.smithi067.stderr:bash: ntpq: command not found 2023-06-23T01:24:19.129 INFO:teuthology.orchestra.run.smithi193.stderr:bash: ntpq: command not found 2023-06-23T01:24:19.131 INFO:teuthology.orchestra.run.smithi067.stdout:210 Number of sources = 4 2023-06-23T01:24:19.131 INFO:teuthology.orchestra.run.smithi067.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:24:19.131 INFO:teuthology.orchestra.run.smithi067.stdout:=============================================================================== 2023-06-23T01:24:19.132 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 +2074us[+2074us] +/- 99ms 2023-06-23T01:24:19.132 INFO:teuthology.orchestra.run.smithi067.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1909us[-1909us] +/- 92ms 2023-06-23T01:24:19.132 INFO:teuthology.orchestra.run.smithi067.stdout:^* hv03.front.sepia.ceph.com 3 6 377 36 +10us[ +17us] +/- 69ms 2023-06-23T01:24:19.132 INFO:teuthology.orchestra.run.smithi067.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:24:19.134 INFO:teuthology.orchestra.run.smithi193.stdout:210 Number of sources = 4 2023-06-23T01:24:19.134 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:24:19.134 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2023-06-23T01:24:19.134 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +2075us[+2084us] +/- 99ms 2023-06-23T01:24:19.135 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1904us[-1896us] +/- 92ms 2023-06-23T01:24:19.135 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv03.front.sepia.ceph.com 3 6 377 33 +5556ns[ +14us] +/- 70ms 2023-06-23T01:24:19.135 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:24:19.135 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-23T01:24:19.149 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-23T01:24:19.150 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-23T01:24:19.189 DEBUG:teuthology.orchestra.run.smithi067:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:24:19.279 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:24:19.365 DEBUG:teuthology.orchestra.run.smithi067:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:24:19.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:24:19.395 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:24:19.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:24:19.425 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-23T01:24:19.434 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-23T01:24:19.465 INFO:teuthology.task.internal:Duration was 1049.159641 seconds 2023-06-23T01:24:19.466 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-23T01:24:19.531 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-23T01:24:19.532 DEBUG:teuthology.orchestra.run.smithi067:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:24:19.534 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:24:19.578 INFO:teuthology.orchestra.run.smithi067.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:24:19.579 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:24:19.979 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-23T01:24:19.980 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi067.front.sepia.ceph.com 2023-06-23T01:24:19.981 DEBUG:teuthology.orchestra.run.smithi067:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:24:20.001 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2023-06-23T01:24:20.002 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:24:20.038 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-23T01:24:20.038 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:24:20.050 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:24:20.115 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-23T01:24:20.116 DEBUG:teuthology.orchestra.run.smithi067:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:24:20.124 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:24:20.224 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-23T01:24:20.245 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-23T01:24:20.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:24:20.271 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:24:20.296 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-23T01:24:20.315 DEBUG:teuthology.orchestra.run.smithi067:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:24:20.318 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:24:20.340 INFO:teuthology.orchestra.run.smithi067.stdout:kernel.core_pattern = core 2023-06-23T01:24:20.360 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2023-06-23T01:24:20.376 DEBUG:teuthology.orchestra.run.smithi067:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:24:20.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:24:20.410 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:24:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:24:20.430 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-23T01:24:20.441 INFO:teuthology.task.internal:Transferring archived files... 2023-06-23T01:24:20.443 DEBUG:teuthology.misc:Transferring archived files from smithi067:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/remote/smithi067 2023-06-23T01:24:20.443 DEBUG:teuthology.orchestra.run.smithi067:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:24:20.485 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945/remote/smithi193 2023-06-23T01:24:20.486 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:24:20.521 INFO:teuthology.task.internal:Removing archive directory... 2023-06-23T01:24:20.521 DEBUG:teuthology.orchestra.run.smithi067:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:24:20.523 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:24:20.568 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-23T01:24:20.584 INFO:teuthology.task.internal:Not uploading archives. 2023-06-23T01:24:20.585 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-23T01:24:20.622 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-23T01:24:20.622 DEBUG:teuthology.orchestra.run.smithi067:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:24:20.625 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:24:20.637 INFO:teuthology.orchestra.run.smithi067.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:24 /home/ubuntu/cephtest 2023-06-23T01:24:20.639 INFO:teuthology.orchestra.run.smithi193.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:24 /home/ubuntu/cephtest 2023-06-23T01:24:20.641 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-23T01:24:20.668 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-23T01:24:20.770 INFO:teuthology.nuke:Checking targets against current locks 2023-06-23T01:24:20.804 DEBUG:teuthology.nuke:shortname: smithi067 2023-06-23T01:24:20.804 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:24:20.831 DEBUG:teuthology.nuke:shortname: smithi193 2023-06-23T01:24:20.832 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:24:20.836 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi067.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:59:10.563533', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDSI9ftZPlRt+IDHUi5hLfCxz7QLJGWQzw8RXkHRFcGsdhLvRaI9dDGsCupQX8AnPEPQNjqvtNkeAzBBPaJrU6mukCrWlcP5De7lGHelE7UA8J1IvH79bsdvgSWpjp3ed5NZVF6A0urzpmxol78o4MUZMibWHiHB+cHw+i18aYXMwKI5H6NkeiqGmD8y0PH5B8rM6vYJg5jVctgUIg/mM+1GV2ZsRKuIQmdiE+6VmHN8aTg4PLF/DWBX2fB2O4j867QZnZUyKaboSKtJgCf8V7FOvYSlewnDvr+yDEwVeXfLxgHR6dLxn+Vl/erM1IIFyqtjJHXASH0J1rOCqENj2g+N5fHaMGGmyTBtjza0h5lVBUSrxkg7BwqzR6xuo/3yoj88cDBrvdcq6f8w6SNcngV35J0dh1x8vERPrjjSn+p10IKVMKkDreofHWSNGsA9CxskVD2dB+40gSsGzQs/ngcwtihMwarIdnHHSAgve3Xt9+vSqrroi44BLzgpooNQss='} 2023-06-23T01:24:20.866 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311945', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:59:10.565178', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBED64u4gjLzpTDT6XA3p9m/odRaKWL8FincvF3djpWK1PElxZKPfNAr+gOw0bHvTOWBCdhDdQlZfrX7eCSWhduQ='} 2023-06-23T01:24:20.895 INFO:teuthology.orchestra.console:Power off smithi067 2023-06-23T01:24:20.897 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:24:20.920 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:24:20.949 INFO:teuthology.orchestra.console:Power off smithi193 2023-06-23T01:24:20.951 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:24:20.978 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:24:24.965 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:24:25.002 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:24:29.080 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi067.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:24:29.117 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:24:29.293 INFO:teuthology.orchestra.console:Power off for smithi067 completed 2023-06-23T01:24:33.236 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:24:33.447 INFO:teuthology.orchestra.console:Power off for smithi193 completed 2023-06-23T01:24:33.553 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/mds-full} duration: 1049.1596405506134 failure_reason: '"2023-06-23T01:23:59.807027+0000 osd.2 (osd.2) 3 : cluster [WRN] OSD bench result of 214026.528160 IOPS exceeded the threshold limit of 500.000000 IOPS for osd.2. IOPS capacity is unchanged at 315.000000 IOPS. The recommendation is to establish the osd''s IOPS capacity using other benchmark tools (e.g. Fio) and then override osd_mclock_max_capacity_iops_[hdd|ssd]." in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=e7c0ca9bff0c40e4aa3fc630bd7b9b5e status: fail success: false 2023-06-23T01:24:33.555 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T01:24:33.640 INFO:teuthology.run:FAIL