2023-06-23T00:47:40.753 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-23T00:47:40.754 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} --verbose --owner scheduled_yuriw@teuthology --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920 -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/orig.config.yaml 2023-06-23T00:47:40.798 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T00:47:40.899 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920 branch: wip-yuri4-testing-2023-06-21-0846-quincy description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} email: yweinste@redhat.com first_in_suite: false job_id: '7311920' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-06-21-0846-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - bad backtrace - object missing on disk - error reading table object - error reading sessionmap - unmatched fragstat - unmatched rstat - was unreadable, recreating it now - Scrub error on inode - Metadata damage detected - inconsistent rstat on inode - Error recovering journal sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c kclient: syntax: v2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-06-21-0846-quincy sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9037 sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2023-06-21-0846-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c targets: smithi039.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA+FEBb0lM2JOjErPpPnTaufA95sbkevpUNdZRQxeWME9cdgBhMzKCyPEvUN+ik/0QBSqca9d17MEbfEIIxoqNw= smithi073.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiGlkx9VorGvQLNcRASuoLUUj4Sv96vdo3OTdVFt9DQkbdF27RlK8Tj2JUas2bUAA8yZ89j7Io0YejSM2nYka4iKTaGqgBVxa4tHUZ+d4PT8GrVn3U/U91xQCGU2+8rgcNAWMPYAC+e4xor6VjopW1+k6DKiC2tc0pL+tMD5EpPA1VVUciLB6I3gSD0Y9Dcjf2mNIzaxnDl+CW6oUcyfq/TGw6tsluKyr14CE6ZELMn1kvUM/zH/P5uQKUiQyhYKzy5JnUzlawpMUQvpCvN1AlqlY1lVyPEogq6bCRl7Rkn+61H4uA85igg/9l29bMuHi4uVHnB+Q02Em6+aVr4/PUnOOyOdvQopTeZm/ZBHYnP5gSo3I+Kk6gr+VGT+lG7F8lBaqzvCCBOWQzLXyuEAuZRwhM5+S2MNEbU5xXPMr3XGp3GrPvCZbzZDmJMqypLlRB7f26kRXilC26fl5ODcmPEK3ecNT+JMwjUxGJCmWiz/BYXrPBZCR4LdJl0OqDqEM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_data_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-22_15:25:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-23T00:47:40.900 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa; will attempt to use it 2023-06-23T00:47:40.901 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks 2023-06-23T00:47:40.901 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-23T00:47:40.903 INFO:teuthology.task.internal:Checking packages... 2023-06-23T00:47:40.932 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '42ddc46e6e6582d8e29a5a982a81886fe028be2c' 2023-06-23T00:47:40.932 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-23T00:47:40.933 INFO:teuthology.packaging:ref: None 2023-06-23T00:47:40.933 INFO:teuthology.packaging:tag: None 2023-06-23T00:47:40.933 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T00:47:40.933 INFO:teuthology.packaging:sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:47:40.934 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T00:47:41.087 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-719.g42ddc46e 2023-06-23T00:47:41.088 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-23T00:47:41.105 INFO:teuthology.task.internal:no buildpackages task found 2023-06-23T00:47:41.106 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-23T00:47:41.164 INFO:teuthology.task.internal:Saving configuration 2023-06-23T00:47:41.184 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-23T00:47:41.197 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T00:47:41.240 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:41:46.876679', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA+FEBb0lM2JOjErPpPnTaufA95sbkevpUNdZRQxeWME9cdgBhMzKCyPEvUN+ik/0QBSqca9d17MEbfEIIxoqNw='} 2023-06-23T00:47:41.264 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:41:46.878554', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOfzMPW2q9RBdkDHsCxaNqsVJBOwqVE0LisPqpvzJYz/gCNLCU/JuCsX3SblAZgE9nIxVTQGHdwvk43mEcp0ylg='} 2023-06-23T00:47:41.265 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-23T00:47:41.271 INFO:teuthology.task.internal:roles: ubuntu@smithi039.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-06-23T00:47:41.272 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-06-23T00:47:41.272 INFO:teuthology.run_tasks:Running task console_log... 2023-06-23T00:47:41.352 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f650b718040>, signals=[15]) 2023-06-23T00:47:41.353 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-23T00:47:41.359 INFO:teuthology.task.internal:Opening connections... 2023-06-23T00:47:41.360 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T00:47:41.362 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:47:41.427 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T00:47:41.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:47:41.500 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-23T00:47:41.507 DEBUG:teuthology.orchestra.run.smithi039:> uname -m 2023-06-23T00:47:41.526 INFO:teuthology.orchestra.run.smithi039.stdout:x86_64 2023-06-23T00:47:41.527 DEBUG:teuthology.orchestra.run.smithi039:> cat /etc/os-release 2023-06-23T00:47:41.583 INFO:teuthology.orchestra.run.smithi039.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T00:47:41.583 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T00:47:41.584 INFO:teuthology.orchestra.run.smithi039.stdout:ID="rhel" 2023-06-23T00:47:41.584 INFO:teuthology.orchestra.run.smithi039.stdout:ID_LIKE="fedora" 2023-06-23T00:47:41.584 INFO:teuthology.orchestra.run.smithi039.stdout:VERSION_ID="8.4" 2023-06-23T00:47:41.584 INFO:teuthology.orchestra.run.smithi039.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:47:41.584 INFO:teuthology.orchestra.run.smithi039.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T00:47:41.584 INFO:teuthology.orchestra.run.smithi039.stdout:ANSI_COLOR="0;31" 2023-06-23T00:47:41.584 INFO:teuthology.orchestra.run.smithi039.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T00:47:41.585 INFO:teuthology.orchestra.run.smithi039.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T00:47:41.585 INFO:teuthology.orchestra.run.smithi039.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T00:47:41.585 INFO:teuthology.orchestra.run.smithi039.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:47:41.585 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T00:47:41.585 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:47:41.585 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T00:47:41.585 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T00:47:41.586 INFO:teuthology.orchestra.run.smithi039.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T00:47:41.586 INFO:teuthology.lock.ops:Updating smithi039.front.sepia.ceph.com on lock server 2023-06-23T00:47:41.619 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2023-06-23T00:47:41.636 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2023-06-23T00:47:41.637 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2023-06-23T00:47:41.693 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T00:47:41.693 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T00:47:41.693 INFO:teuthology.orchestra.run.smithi073.stdout:ID="rhel" 2023-06-23T00:47:41.693 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="fedora" 2023-06-23T00:47:41.694 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="8.4" 2023-06-23T00:47:41.694 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:47:41.694 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T00:47:41.694 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2023-06-23T00:47:41.694 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T00:47:41.695 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T00:47:41.695 INFO:teuthology.orchestra.run.smithi073.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T00:47:41.695 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:47:41.695 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T00:47:41.695 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:47:41.696 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T00:47:41.696 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T00:47:41.696 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T00:47:41.696 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2023-06-23T00:47:41.727 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-23T00:47:41.738 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-23T00:47:41.749 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-23T00:47:41.749 DEBUG:teuthology.orchestra.run.smithi039:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:47:41.752 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:47:41.768 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-23T00:47:41.775 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-23T00:47:41.775 DEBUG:teuthology.orchestra.run.smithi039:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:47:41.808 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:47:41.823 INFO:teuthology.orchestra.run.smithi039.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T00:47:41.824 INFO:teuthology.orchestra.run.smithi073.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T00:47:41.825 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-23T00:47:41.885 INFO:teuthology.run_tasks:Running task kernel... 2023-06-23T00:47:41.903 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-06-23T00:47:41.905 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-06-23T00:47:41.905 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-06-23T00:47:41.905 DEBUG:teuthology.orchestra.run.smithi039:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:47:41.906 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:47:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:47:41.922 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2023-06-23T00:47:41.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:47:41.924 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-06-23T00:47:41.979 INFO:teuthology.orchestra.run.smithi073.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:47:41.979 INFO:teuthology.task.kernel:Running kernel on smithi073: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:47:41.979 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum install -y kernel 2023-06-23T00:47:41.981 INFO:teuthology.orchestra.run.smithi039.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:47:41.982 INFO:teuthology.task.kernel:Running kernel on smithi039: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T00:47:41.982 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum install -y kernel 2023-06-23T00:47:43.358 INFO:teuthology.orchestra.run.smithi073.stdout:Updating Subscription Management repositories. 2023-06-23T00:47:43.359 INFO:teuthology.orchestra.run.smithi073.stdout:Unable to read consumer identity 2023-06-23T00:47:43.439 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-23T00:47:43.440 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-06-23T00:47:47.002 INFO:teuthology.orchestra.run.smithi073.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-06-23T00:47:47.286 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T00:47:49.236 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 7.1 MB/s | 16 MB 00:02 2023-06-23T00:47:49.392 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 7.6 MB/s | 16 MB 00:02 2023-06-23T00:47:52.490 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-06-23T00:47:52.785 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 4.5 MB/s | 24 kB 00:00 2023-06-23T00:47:53.232 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T00:47:53.233 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T00:47:53.233 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T00:47:53.378 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-06-23T00:47:53.379 INFO:teuthology.orchestra.run.smithi073.stdout:Nothing to do. 2023-06-23T00:47:53.379 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-06-23T00:47:53.492 DEBUG:teuthology.orchestra.run.smithi073:> echo no | sudo yum reinstall kernel || true 2023-06-23T00:47:53.521 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T00:47:53.522 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T00:47:53.522 INFO:teuthology.orchestra.run.smithi039.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T00:47:53.689 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-06-23T00:47:53.690 INFO:teuthology.orchestra.run.smithi039.stdout:Nothing to do. 2023-06-23T00:47:53.690 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-06-23T00:47:53.807 DEBUG:teuthology.orchestra.run.smithi039:> echo no | sudo yum reinstall kernel || true 2023-06-23T00:47:53.872 INFO:teuthology.orchestra.run.smithi073.stdout:Updating Subscription Management repositories. 2023-06-23T00:47:53.873 INFO:teuthology.orchestra.run.smithi073.stdout:Unable to read consumer identity 2023-06-23T00:47:54.163 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 12:47:52 AM UTC. 2023-06-23T00:47:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-23T00:47:54.171 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-06-23T00:47:54.293 INFO:teuthology.orchestra.run.smithi073.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:47:54.293 INFO:teuthology.orchestra.run.smithi073.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:47:54.293 INFO:teuthology.orchestra.run.smithi073.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:47:54.317 INFO:teuthology.orchestra.run.smithi073.stderr:Error: No packages marked for reinstall. 2023-06-23T00:47:54.366 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum reinstall -y kernel || true 2023-06-23T00:47:54.474 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 12:47:52 AM UTC. 2023-06-23T00:47:54.616 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:47:54.616 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:47:54.616 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:47:54.641 INFO:teuthology.orchestra.run.smithi039.stderr:Error: No packages marked for reinstall. 2023-06-23T00:47:54.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum reinstall -y kernel || true 2023-06-23T00:47:54.708 INFO:teuthology.orchestra.run.smithi073.stdout:Updating Subscription Management repositories. 2023-06-23T00:47:54.708 INFO:teuthology.orchestra.run.smithi073.stdout:Unable to read consumer identity 2023-06-23T00:47:54.989 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 12:47:52 AM UTC. 2023-06-23T00:47:55.057 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-23T00:47:55.057 INFO:teuthology.orchestra.run.smithi039.stdout:Unable to read consumer identity 2023-06-23T00:47:55.125 INFO:teuthology.orchestra.run.smithi073.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:47:55.125 INFO:teuthology.orchestra.run.smithi073.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:47:55.125 INFO:teuthology.orchestra.run.smithi073.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:47:55.139 INFO:teuthology.orchestra.run.smithi073.stderr:Error: No packages marked for reinstall. 2023-06-23T00:47:55.189 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:47:55.313 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.314 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.314 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.314 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T00:47:55.314 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T00:47:55.315 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-06-23T00:47:55.315 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi073.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T00:47:55.315 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:47:55.356 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:00:03 ago on Fri 23 Jun 2023 12:47:52 AM UTC. 2023-06-23T00:47:55.440 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.441 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.441 DEBUG:teuthology.orchestra.run.smithi073:> sudo rpm -qi grub2-tools 2023-06-23T00:47:55.468 INFO:teuthology.orchestra.run.smithi073.stdout:Name : grub2-tools 2023-06-23T00:47:55.468 INFO:teuthology.orchestra.run.smithi073.stdout:Epoch : 1 2023-06-23T00:47:55.469 INFO:teuthology.orchestra.run.smithi073.stdout:Version : 2.02 2023-06-23T00:47:55.469 INFO:teuthology.orchestra.run.smithi073.stdout:Release : 99.el8_4.1 2023-06-23T00:47:55.469 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: x86_64 2023-06-23T00:47:55.469 INFO:teuthology.orchestra.run.smithi073.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T00:47:55.469 INFO:teuthology.orchestra.run.smithi073.stdout:Group : System Environment/Base 2023-06-23T00:47:55.469 INFO:teuthology.orchestra.run.smithi073.stdout:Size : 9264631 2023-06-23T00:47:55.469 INFO:teuthology.orchestra.run.smithi073.stdout:License : GPLv3+ 2023-06-23T00:47:55.470 INFO:teuthology.orchestra.run.smithi073.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T00:47:55.470 INFO:teuthology.orchestra.run.smithi073.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T00:47:55.470 INFO:teuthology.orchestra.run.smithi073.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T00:47:55.470 INFO:teuthology.orchestra.run.smithi073.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T00:47:55.470 INFO:teuthology.orchestra.run.smithi073.stdout:Relocations : (not relocatable) 2023-06-23T00:47:55.470 INFO:teuthology.orchestra.run.smithi073.stdout:Packager : Red Hat, Inc. 2023-06-23T00:47:55.470 INFO:teuthology.orchestra.run.smithi073.stdout:Vendor : Red Hat, Inc. 2023-06-23T00:47:55.470 INFO:teuthology.orchestra.run.smithi073.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T00:47:55.471 INFO:teuthology.orchestra.run.smithi073.stdout:Summary : Support tools for GRUB. 2023-06-23T00:47:55.471 INFO:teuthology.orchestra.run.smithi073.stdout:Description : 2023-06-23T00:47:55.471 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T00:47:55.471 INFO:teuthology.orchestra.run.smithi073.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T00:47:55.471 INFO:teuthology.orchestra.run.smithi073.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T00:47:55.471 INFO:teuthology.orchestra.run.smithi073.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T00:47:55.471 INFO:teuthology.orchestra.run.smithi073.stdout:hardware devices. 2023-06-23T00:47:55.472 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T00:47:55.472 INFO:teuthology.orchestra.run.smithi073.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T00:47:55.473 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T00:47:55.473 INFO:teuthology.task.kernel:Updating grub on smithi073 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.473 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T00:47:55.574 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T00:47:55.574 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:47:55.574 INFO:teuthology.orchestra.run.smithi039.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T00:47:55.583 INFO:teuthology.orchestra.run.smithi039.stderr:Error: No packages marked for reinstall. 2023-06-23T00:47:55.639 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:47:55.770 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.771 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.771 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.771 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T00:47:55.771 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T00:47:55.771 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-06-23T00:47:55.772 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi039.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T00:47:55.772 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T00:47:55.903 INFO:teuthology.orchestra.run.smithi039.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.903 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.904 DEBUG:teuthology.orchestra.run.smithi039:> sudo rpm -qi grub2-tools 2023-06-23T00:47:55.933 INFO:teuthology.orchestra.run.smithi039.stdout:Name : grub2-tools 2023-06-23T00:47:55.933 INFO:teuthology.orchestra.run.smithi039.stdout:Epoch : 1 2023-06-23T00:47:55.933 INFO:teuthology.orchestra.run.smithi039.stdout:Version : 2.02 2023-06-23T00:47:55.933 INFO:teuthology.orchestra.run.smithi039.stdout:Release : 99.el8_4.1 2023-06-23T00:47:55.934 INFO:teuthology.orchestra.run.smithi039.stdout:Architecture: x86_64 2023-06-23T00:47:55.934 INFO:teuthology.orchestra.run.smithi039.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T00:47:55.934 INFO:teuthology.orchestra.run.smithi039.stdout:Group : System Environment/Base 2023-06-23T00:47:55.934 INFO:teuthology.orchestra.run.smithi039.stdout:Size : 9264631 2023-06-23T00:47:55.934 INFO:teuthology.orchestra.run.smithi039.stdout:License : GPLv3+ 2023-06-23T00:47:55.934 INFO:teuthology.orchestra.run.smithi039.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T00:47:55.934 INFO:teuthology.orchestra.run.smithi039.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T00:47:55.935 INFO:teuthology.orchestra.run.smithi039.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T00:47:55.935 INFO:teuthology.orchestra.run.smithi039.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T00:47:55.935 INFO:teuthology.orchestra.run.smithi039.stdout:Relocations : (not relocatable) 2023-06-23T00:47:55.935 INFO:teuthology.orchestra.run.smithi039.stdout:Packager : Red Hat, Inc. 2023-06-23T00:47:55.935 INFO:teuthology.orchestra.run.smithi039.stdout:Vendor : Red Hat, Inc. 2023-06-23T00:47:55.935 INFO:teuthology.orchestra.run.smithi039.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T00:47:55.935 INFO:teuthology.orchestra.run.smithi039.stdout:Summary : Support tools for GRUB. 2023-06-23T00:47:55.936 INFO:teuthology.orchestra.run.smithi039.stdout:Description : 2023-06-23T00:47:55.936 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T00:47:55.936 INFO:teuthology.orchestra.run.smithi039.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T00:47:55.936 INFO:teuthology.orchestra.run.smithi039.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T00:47:55.936 INFO:teuthology.orchestra.run.smithi039.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T00:47:55.937 INFO:teuthology.orchestra.run.smithi039.stdout:hardware devices. 2023-06-23T00:47:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T00:47:55.937 INFO:teuthology.orchestra.run.smithi039.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T00:47:55.938 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T00:47:55.938 INFO:teuthology.task.kernel:Updating grub on smithi039 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:55.938 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T00:47:56.505 INFO:teuthology.orchestra.run.smithi073.stderr:Generating grub configuration file ... 2023-06-23T00:47:57.250 INFO:teuthology.orchestra.run.smithi039.stderr:Generating grub configuration file ... 2023-06-23T00:47:58.825 INFO:teuthology.orchestra.run.smithi073.stderr:done 2023-06-23T00:47:58.826 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2023-06-23T00:47:58.841 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.P4l8Zx6xBl 2023-06-23T00:47:58.841 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.P4l8Zx6xBl 2023-06-23T00:47:58.919 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.P4l8Zx6xBl 2023-06-23T00:47:59.076 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.P4l8Zx6xBl is 6KB 2023-06-23T00:47:59.125 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.P4l8Zx6xBl 2023-06-23T00:47:59.138 DEBUG:teuthology.orchestra.run.smithi073:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T00:47:59.201 INFO:teuthology.orchestra.run.smithi073.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T00:47:59.201 INFO:teuthology.orchestra.run.smithi073.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T00:47:59.202 INFO:teuthology.orchestra.run.smithi073.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T00:47:59.202 INFO:teuthology.orchestra.run.smithi073.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T00:47:59.203 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:59.371 DEBUG:teuthology.orchestra.run.smithi073:> sudo shutdown -r now 2023-06-23T00:47:59.422 INFO:teuthology.orchestra.run.smithi039.stderr:done 2023-06-23T00:47:59.424 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-06-23T00:47:59.447 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.TzodlJ1o6C 2023-06-23T00:47:59.447 DEBUG:teuthology.orchestra.run.smithi039:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TzodlJ1o6C 2023-06-23T00:47:59.516 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.TzodlJ1o6C 2023-06-23T00:47:59.675 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.TzodlJ1o6C is 6KB 2023-06-23T00:47:59.723 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.TzodlJ1o6C 2023-06-23T00:47:59.737 DEBUG:teuthology.orchestra.run.smithi039:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T00:47:59.819 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T00:47:59.819 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T00:47:59.820 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T00:47:59.820 INFO:teuthology.orchestra.run.smithi039.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T00:47:59.821 DEBUG:teuthology.orchestra.run.smithi039:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T00:47:59.927 DEBUG:teuthology.orchestra.run.smithi039:> sudo shutdown -r now 2023-06-23T00:48:29.403 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-06-23T00:48:29.404 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-06-23T00:48:29.404 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:48:29.931 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-06-23T00:48:29.931 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-23T00:48:29.932 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:48:47.728 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2023-06-23T00:49:04.628 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.39 2023-06-23T00:49:17.740 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-06-23T00:49:17.741 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:49:34.646 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-23T00:49:34.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:49:36.112 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2023-06-23T00:50:06.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-06-23T00:50:06.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:50:06.528 DEBUG:teuthology.orchestra.run.smithi073:> true 2023-06-23T00:50:06.838 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-06-23T00:50:06.838 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T00:50:06.839 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2023-06-23T00:50:06.855 INFO:teuthology.orchestra.run.smithi073.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T00:50:06.856 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:50:06.856 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T00:50:06.857 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T00:50:07.858 DEBUG:teuthology.parallel:result is None 2023-06-23T00:50:34.649 DEBUG:teuthology.orchestra.remote:timed out 2023-06-23T00:51:04.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-23T00:51:04.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:51:04.958 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-06-23T00:51:05.204 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-23T00:51:05.204 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T00:51:05.204 DEBUG:teuthology.orchestra.run.smithi039:> uname -r 2023-06-23T00:51:05.220 INFO:teuthology.orchestra.run.smithi039.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T00:51:05.220 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T00:51:05.220 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T00:51:05.221 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T00:51:06.222 DEBUG:teuthology.parallel:result is None 2023-06-23T00:51:06.222 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-23T00:51:06.257 INFO:teuthology.task.internal:Creating test directory... 2023-06-23T00:51:06.257 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T00:51:06.260 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T00:51:06.277 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-23T00:51:06.326 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-23T00:51:06.354 INFO:teuthology.task.internal:Creating archive directory... 2023-06-23T00:51:06.355 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T00:51:06.357 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T00:51:06.393 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-23T00:51:06.403 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-23T00:51:06.404 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T00:51:06.423 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T00:51:06.503 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:51:06.520 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:51:06.522 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:51:06.537 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T00:51:06.538 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-23T00:51:06.549 INFO:teuthology.task.internal:Configuring sudo... 2023-06-23T00:51:06.549 DEBUG:teuthology.orchestra.run.smithi039:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T00:51:06.552 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T00:51:06.578 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-23T00:51:06.590 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-23T00:51:06.590 DEBUG:teuthology.orchestra.run.smithi039:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T00:51:06.621 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T00:51:06.636 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:51:06.723 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:51:06.782 DEBUG:teuthology.orchestra.run.smithi039:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:51:06.826 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:51:06.867 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T00:51:06.867 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T00:51:06.933 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:51:06.977 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T00:51:07.018 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:51:07.102 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T00:51:07.140 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T00:51:07.140 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T00:51:07.204 DEBUG:teuthology.orchestra.run.smithi039:> sudo service rsyslog restart 2023-06-23T00:51:07.207 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2023-06-23T00:51:07.257 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:51:07.299 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T00:51:07.613 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-23T00:51:07.627 INFO:teuthology.task.internal:Starting timer... 2023-06-23T00:51:07.627 INFO:teuthology.run_tasks:Running task pcp... 2023-06-23T00:51:07.663 INFO:teuthology.run_tasks:Running task selinux... 2023-06-23T00:51:07.683 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-06-23T00:51:07.683 DEBUG:teuthology.orchestra.run.smithi039:> sudo service auditd rotate 2023-06-23T00:51:07.736 INFO:teuthology.orchestra.run.smithi039.stdout:Rotating logs: [ OK ] 2023-06-23T00:51:07.738 DEBUG:teuthology.orchestra.run.smithi073:> sudo service auditd rotate 2023-06-23T00:51:07.796 INFO:teuthology.orchestra.run.smithi073.stdout:Rotating logs: [ OK ] 2023-06-23T00:51:07.799 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-23T00:51:07.799 DEBUG:teuthology.orchestra.run.smithi039:> /usr/sbin/getenforce 2023-06-23T00:51:07.834 INFO:teuthology.orchestra.run.smithi039.stdout:Permissive 2023-06-23T00:51:07.835 DEBUG:teuthology.orchestra.run.smithi073:> /usr/sbin/getenforce 2023-06-23T00:51:07.863 INFO:teuthology.orchestra.run.smithi073.stdout:Permissive 2023-06-23T00:51:07.863 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi039.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi073.front.sepia.ceph.com': 'permissive'} 2023-06-23T00:51:07.864 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:51:07.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:51:07.901 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T00:51:07.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:51:07.930 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-23T00:51:07.930 DEBUG:teuthology.orchestra.run.smithi039:> sudo /usr/sbin/setenforce permissive 2023-06-23T00:51:07.974 DEBUG:teuthology.orchestra.run.smithi073:> sudo /usr/sbin/setenforce permissive 2023-06-23T00:51:08.000 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-23T00:51:08.012 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-06-23T00:51:08.013 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-23T00:51:08.041 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-23T00:51:08.043 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi039.front.sepia.ceph.com,smithi073.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-23T01:05:50.722 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi039.front.sepia.ceph.com'), Remote(name='ubuntu@smithi073.front.sepia.ceph.com')] 2023-06-23T01:05:50.724 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi039.front.sepia.ceph.com' 2023-06-23T01:05:50.725 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi039.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:05:50.794 DEBUG:teuthology.orchestra.run.smithi039:> true 2023-06-23T01:05:50.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2023-06-23T01:05:50.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:05:50.926 DEBUG:teuthology.orchestra.run.smithi073:> true 2023-06-23T01:05:50.990 INFO:teuthology.run_tasks:Running task clock... 2023-06-23T01:05:51.006 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-23T01:05:51.007 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T01:05:51.007 DEBUG:teuthology.orchestra.run.smithi039:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:05:51.009 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T01:05:51.010 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:05:51.038 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T01:05:51.053 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T01:05:51.058 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T01:05:51.072 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T01:05:51.090 INFO:teuthology.orchestra.run.smithi039.stderr:sudo: ntpd: command not found 2023-06-23T01:05:51.100 INFO:teuthology.orchestra.run.smithi039.stdout:506 Cannot talk to daemon 2023-06-23T01:05:51.110 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: ntpd: command not found 2023-06-23T01:05:51.114 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T01:05:51.121 INFO:teuthology.orchestra.run.smithi073.stdout:506 Cannot talk to daemon 2023-06-23T01:05:51.128 INFO:teuthology.orchestra.run.smithi039.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T01:05:51.134 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T01:05:51.148 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T01:05:51.201 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-06-23T01:05:51.204 INFO:teuthology.orchestra.run.smithi039.stdout:210 Number of sources = 4 2023-06-23T01:05:51.204 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:05:51.204 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-06-23T01:05:51.204 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:05:51.204 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:05:51.204 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:05:51.205 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:05:51.215 INFO:teuthology.orchestra.run.smithi073.stderr:bash: ntpq: command not found 2023-06-23T01:05:51.217 INFO:teuthology.orchestra.run.smithi073.stdout:210 Number of sources = 4 2023-06-23T01:05:51.217 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:05:51.217 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2023-06-23T01:05:51.218 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:05:51.218 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:05:51.218 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:05:51.218 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:05:51.219 INFO:teuthology.run_tasks:Running task install... 2023-06-23T01:05:51.227 DEBUG:teuthology.task.install:project ceph 2023-06-23T01:05:51.228 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}} 2023-06-23T01:05:51.228 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-23T01:05:51.228 INFO:teuthology.task.install:Using flavor: default 2023-06-23T01:05:51.233 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-23T01:05:51.234 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-23T01:05:51.234 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T01:05:51.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:05:51.237 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T01:05:51.238 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:05:51.381 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T01:05:51.381 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T01:05:51.385 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T01:05:51.385 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T01:05:51.607 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T01:05:51.607 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:05:51.607 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T01:05:51.611 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T01:05:51.612 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:05:51.612 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T01:05:51.633 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T01:05:51.633 DEBUG:teuthology.orchestra.run.smithi073:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T01:05:51.638 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T01:05:51.638 DEBUG:teuthology.orchestra.run.smithi039:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T01:05:51.698 DEBUG:teuthology.orchestra.run.smithi073:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T01:05:51.704 DEBUG:teuthology.orchestra.run.smithi039:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T01:05:51.773 DEBUG:teuthology.orchestra.run.smithi073:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T01:05:51.780 DEBUG:teuthology.orchestra.run.smithi039:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T01:05:51.836 INFO:teuthology.orchestra.run.smithi073.stdout:check_obsoletes = 1 2023-06-23T01:05:51.838 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum clean all 2023-06-23T01:05:51.845 INFO:teuthology.orchestra.run.smithi039.stdout:check_obsoletes = 1 2023-06-23T01:05:51.846 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum clean all 2023-06-23T01:05:52.227 INFO:teuthology.orchestra.run.smithi073.stdout:Updating Subscription Management repositories. 2023-06-23T01:05:52.255 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-23T01:05:52.532 INFO:teuthology.orchestra.run.smithi073.stdout:44 files removed 2023-06-23T01:05:52.581 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T01:05:52.583 INFO:teuthology.orchestra.run.smithi039.stdout:44 files removed 2023-06-23T01:05:52.634 DEBUG:teuthology.orchestra.run.smithi039:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T01:05:52.930 INFO:teuthology.orchestra.run.smithi073.stdout:Updating Subscription Management repositories. 2023-06-23T01:05:53.002 INFO:teuthology.orchestra.run.smithi039.stdout:Updating Subscription Management repositories. 2023-06-23T01:05:53.695 INFO:teuthology.orchestra.run.smithi073.stdout:ceph packages for x86_64 279 kB/s | 78 kB 00:00 2023-06-23T01:05:53.785 INFO:teuthology.orchestra.run.smithi039.stdout:ceph packages for x86_64 251 kB/s | 78 kB 00:00 2023-06-23T01:05:53.981 INFO:teuthology.orchestra.run.smithi073.stdout:ceph noarch packages 37 kB/s | 9.2 kB 00:00 2023-06-23T01:05:54.135 INFO:teuthology.orchestra.run.smithi039.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-06-23T01:05:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:ceph source packages 4.8 kB/s | 1.4 kB 00:00 2023-06-23T01:05:54.408 INFO:teuthology.orchestra.run.smithi039.stdout:ceph source packages 6.0 kB/s | 1.4 kB 00:00 2023-06-23T01:05:56.557 INFO:teuthology.orchestra.run.smithi073.stdout:Copr repo for python3-asyncssh owned by ceph 1.6 kB/s | 3.5 kB 00:02 2023-06-23T01:05:56.879 INFO:teuthology.orchestra.run.smithi039.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T01:05:57.881 INFO:teuthology.orchestra.run.smithi039.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:00 2023-06-23T01:05:59.005 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 6.5 MB/s | 16 MB 00:02 2023-06-23T01:06:01.268 INFO:teuthology.orchestra.run.smithi039.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-06-23T01:06:01.658 INFO:teuthology.orchestra.run.smithi039.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 108 MB/s | 33 MB 00:00 2023-06-23T01:06:02.239 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2023-06-23T01:06:02.630 INFO:teuthology.orchestra.run.smithi073.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 123 MB/s | 38 MB 00:00 2023-06-23T01:06:07.140 INFO:teuthology.orchestra.run.smithi039.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 112 MB/s | 38 MB 00:00 2023-06-23T01:06:08.125 INFO:teuthology.orchestra.run.smithi073.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 MB/s | 33 MB 00:00 2023-06-23T01:06:15.971 INFO:teuthology.orchestra.run.smithi073.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:06:15.972 INFO:teuthology.orchestra.run.smithi073.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:06:15.983 INFO:teuthology.orchestra.run.smithi073.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T01:06:15.983 INFO:teuthology.orchestra.run.smithi073.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T01:06:15.984 INFO:teuthology.orchestra.run.smithi073.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T01:06:15.985 INFO:teuthology.orchestra.run.smithi073.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T01:06:15.987 INFO:teuthology.orchestra.run.smithi073.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T01:06:15.989 INFO:teuthology.orchestra.run.smithi073.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T01:06:15.990 INFO:teuthology.orchestra.run.smithi073.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T01:06:16.066 INFO:teuthology.orchestra.run.smithi039.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:06:16.067 INFO:teuthology.orchestra.run.smithi039.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:06:16.078 INFO:teuthology.orchestra.run.smithi039.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T01:06:16.079 INFO:teuthology.orchestra.run.smithi039.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T01:06:16.080 INFO:teuthology.orchestra.run.smithi039.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T01:06:16.080 INFO:teuthology.orchestra.run.smithi039.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T01:06:16.082 INFO:teuthology.orchestra.run.smithi039.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T01:06:16.085 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T01:06:16.085 INFO:teuthology.orchestra.run.smithi039.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T01:06:16.488 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2023-06-23T01:06:16.499 INFO:teuthology.orchestra.run.smithi073.stdout:====================================================================================================================== 2023-06-23T01:06:16.500 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2023-06-23T01:06:16.500 INFO:teuthology.orchestra.run.smithi073.stdout:====================================================================================================================== 2023-06-23T01:06:16.500 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2023-06-23T01:06:16.500 INFO:teuthology.orchestra.run.smithi073.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T01:06:16.500 INFO:teuthology.orchestra.run.smithi073.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T01:06:16.500 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T01:06:16.501 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T01:06:16.501 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T01:06:16.501 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T01:06:16.501 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T01:06:16.501 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T01:06:16.501 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T01:06:16.501 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T01:06:16.502 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:06:16.502 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T01:06:16.502 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T01:06:16.502 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T01:06:16.502 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T01:06:16.502 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T01:06:16.502 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T01:06:16.503 INFO:teuthology.orchestra.run.smithi073.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T01:06:16.503 INFO:teuthology.orchestra.run.smithi073.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T01:06:16.503 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T01:06:16.503 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T01:06:16.503 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T01:06:16.503 INFO:teuthology.orchestra.run.smithi073.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T01:06:16.503 INFO:teuthology.orchestra.run.smithi073.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T01:06:16.503 INFO:teuthology.orchestra.run.smithi073.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T01:06:16.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T01:06:16.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T01:06:16.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T01:06:16.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T01:06:16.504 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T01:06:16.504 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:06:16.504 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T01:06:16.505 INFO:teuthology.orchestra.run.smithi073.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T01:06:16.505 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrading: 2023-06-23T01:06:16.505 INFO:teuthology.orchestra.run.smithi073.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:06:16.505 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T01:06:16.506 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2023-06-23T01:06:16.506 INFO:teuthology.orchestra.run.smithi073.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T01:06:16.506 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T01:06:16.506 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T01:06:16.506 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T01:06:16.506 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T01:06:16.507 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T01:06:16.507 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:06:16.507 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T01:06:16.507 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T01:06:16.507 INFO:teuthology.orchestra.run.smithi073.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T01:06:16.507 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T01:06:16.507 INFO:teuthology.orchestra.run.smithi073.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T01:06:16.508 INFO:teuthology.orchestra.run.smithi073.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T01:06:16.508 INFO:teuthology.orchestra.run.smithi073.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T01:06:16.508 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T01:06:16.508 INFO:teuthology.orchestra.run.smithi073.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T01:06:16.508 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T01:06:16.508 INFO:teuthology.orchestra.run.smithi073.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T01:06:16.508 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T01:06:16.509 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T01:06:16.509 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T01:06:16.509 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T01:06:16.509 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T01:06:16.509 INFO:teuthology.orchestra.run.smithi073.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T01:06:16.509 INFO:teuthology.orchestra.run.smithi073.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T01:06:16.509 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T01:06:16.510 INFO:teuthology.orchestra.run.smithi073.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T01:06:16.510 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T01:06:16.510 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T01:06:16.510 INFO:teuthology.orchestra.run.smithi073.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T01:06:16.510 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T01:06:16.510 INFO:teuthology.orchestra.run.smithi073.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T01:06:16.510 INFO:teuthology.orchestra.run.smithi073.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T01:06:16.511 INFO:teuthology.orchestra.run.smithi073.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T01:06:16.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T01:06:16.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T01:06:16.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T01:06:16.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T01:06:16.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T01:06:16.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T01:06:16.511 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T01:06:16.512 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T01:06:16.512 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T01:06:16.512 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T01:06:16.512 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T01:06:16.512 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T01:06:16.512 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T01:06:16.513 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T01:06:16.513 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T01:06:16.513 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T01:06:16.513 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:06:16.513 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T01:06:16.513 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T01:06:16.513 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T01:06:16.514 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T01:06:16.514 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T01:06:16.514 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T01:06:16.514 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T01:06:16.514 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T01:06:16.514 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T01:06:16.515 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T01:06:16.515 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T01:06:16.515 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T01:06:16.515 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T01:06:16.515 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T01:06:16.516 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T01:06:16.516 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T01:06:16.516 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:06:16.516 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T01:06:16.516 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T01:06:16.516 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T01:06:16.516 INFO:teuthology.orchestra.run.smithi073.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T01:06:16.517 INFO:teuthology.orchestra.run.smithi073.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T01:06:16.517 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T01:06:16.517 INFO:teuthology.orchestra.run.smithi073.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T01:06:16.517 INFO:teuthology.orchestra.run.smithi073.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T01:06:16.517 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T01:06:16.518 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T01:06:16.518 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T01:06:16.518 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T01:06:16.518 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T01:06:16.518 INFO:teuthology.orchestra.run.smithi073.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T01:06:16.518 INFO:teuthology.orchestra.run.smithi073.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T01:06:16.518 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T01:06:16.519 INFO:teuthology.orchestra.run.smithi073.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T01:06:16.519 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:06:16.519 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2023-06-23T01:06:16.519 INFO:teuthology.orchestra.run.smithi073.stdout:====================================================================================================================== 2023-06-23T01:06:16.519 INFO:teuthology.orchestra.run.smithi073.stdout:Install 118 Packages 2023-06-23T01:06:16.519 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrade 2 Packages 2023-06-23T01:06:16.519 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:06:16.520 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 201 M 2023-06-23T01:06:16.521 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2023-06-23T01:06:16.598 INFO:teuthology.orchestra.run.smithi039.stdout:Dependencies resolved. 2023-06-23T01:06:16.610 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-06-23T01:06:16.610 INFO:teuthology.orchestra.run.smithi039.stdout: Package Arch Version Repository Size 2023-06-23T01:06:16.610 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-06-23T01:06:16.611 INFO:teuthology.orchestra.run.smithi039.stdout:Installing: 2023-06-23T01:06:16.611 INFO:teuthology.orchestra.run.smithi039.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T01:06:16.611 INFO:teuthology.orchestra.run.smithi039.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T01:06:16.611 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T01:06:16.611 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T01:06:16.611 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T01:06:16.611 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T01:06:16.612 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T01:06:16.612 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T01:06:16.612 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T01:06:16.612 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T01:06:16.612 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:06:16.612 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T01:06:16.612 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T01:06:16.613 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T01:06:16.613 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T01:06:16.613 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T01:06:16.613 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T01:06:16.613 INFO:teuthology.orchestra.run.smithi039.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T01:06:16.613 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T01:06:16.614 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T01:06:16.614 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T01:06:16.614 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T01:06:16.614 INFO:teuthology.orchestra.run.smithi039.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T01:06:16.614 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T01:06:16.614 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T01:06:16.614 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T01:06:16.615 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T01:06:16.615 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T01:06:16.615 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T01:06:16.615 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T01:06:16.615 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:06:16.615 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T01:06:16.615 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T01:06:16.615 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrading: 2023-06-23T01:06:16.616 INFO:teuthology.orchestra.run.smithi039.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:06:16.616 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T01:06:16.616 INFO:teuthology.orchestra.run.smithi039.stdout:Installing dependencies: 2023-06-23T01:06:16.616 INFO:teuthology.orchestra.run.smithi039.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T01:06:16.616 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T01:06:16.616 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T01:06:16.616 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T01:06:16.617 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T01:06:16.617 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T01:06:16.617 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:06:16.617 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T01:06:16.617 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T01:06:16.617 INFO:teuthology.orchestra.run.smithi039.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T01:06:16.617 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T01:06:16.617 INFO:teuthology.orchestra.run.smithi039.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T01:06:16.618 INFO:teuthology.orchestra.run.smithi039.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T01:06:16.618 INFO:teuthology.orchestra.run.smithi039.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T01:06:16.618 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T01:06:16.618 INFO:teuthology.orchestra.run.smithi039.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T01:06:16.618 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T01:06:16.618 INFO:teuthology.orchestra.run.smithi039.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T01:06:16.619 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T01:06:16.619 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T01:06:16.619 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T01:06:16.619 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T01:06:16.619 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T01:06:16.619 INFO:teuthology.orchestra.run.smithi039.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T01:06:16.620 INFO:teuthology.orchestra.run.smithi039.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T01:06:16.620 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T01:06:16.620 INFO:teuthology.orchestra.run.smithi039.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T01:06:16.620 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T01:06:16.620 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T01:06:16.621 INFO:teuthology.orchestra.run.smithi039.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T01:06:16.621 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T01:06:16.621 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T01:06:16.621 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T01:06:16.622 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T01:06:16.622 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T01:06:16.622 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T01:06:16.622 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T01:06:16.622 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T01:06:16.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T01:06:16.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T01:06:16.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T01:06:16.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T01:06:16.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T01:06:16.623 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T01:06:16.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T01:06:16.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T01:06:16.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T01:06:16.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T01:06:16.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T01:06:16.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T01:06:16.624 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:06:16.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T01:06:16.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T01:06:16.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T01:06:16.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T01:06:16.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T01:06:16.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T01:06:16.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T01:06:16.625 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T01:06:16.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T01:06:16.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T01:06:16.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T01:06:16.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T01:06:16.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T01:06:16.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T01:06:16.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T01:06:16.626 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T01:06:16.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:06:16.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T01:06:16.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T01:06:16.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T01:06:16.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T01:06:16.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T01:06:16.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T01:06:16.627 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T01:06:16.628 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T01:06:16.628 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T01:06:16.628 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T01:06:16.628 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T01:06:16.628 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T01:06:16.628 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T01:06:16.628 INFO:teuthology.orchestra.run.smithi039.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T01:06:16.629 INFO:teuthology.orchestra.run.smithi039.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T01:06:16.629 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T01:06:16.629 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T01:06:16.629 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:06:16.629 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction Summary 2023-06-23T01:06:16.629 INFO:teuthology.orchestra.run.smithi039.stdout:====================================================================================================================== 2023-06-23T01:06:16.629 INFO:teuthology.orchestra.run.smithi039.stdout:Install 118 Packages 2023-06-23T01:06:16.629 INFO:teuthology.orchestra.run.smithi039.stdout:Upgrade 2 Packages 2023-06-23T01:06:16.630 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:06:16.630 INFO:teuthology.orchestra.run.smithi039.stdout:Total download size: 201 M 2023-06-23T01:06:16.630 INFO:teuthology.orchestra.run.smithi039.stdout:Downloading Packages: 2023-06-23T01:06:16.909 INFO:teuthology.orchestra.run.smithi073.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 28 kB/s | 6.4 kB 00:00 2023-06-23T01:06:17.116 INFO:teuthology.orchestra.run.smithi039.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 20 kB/s | 6.4 kB 00:00 2023-06-23T01:06:17.421 INFO:teuthology.orchestra.run.smithi073.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.7 MB/s | 873 kB 00:00 2023-06-23T01:06:17.496 INFO:teuthology.orchestra.run.smithi073.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 2.1 MB/s | 162 kB 00:00 2023-06-23T01:06:17.803 INFO:teuthology.orchestra.run.smithi039.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.2 MB/s | 873 kB 00:00 2023-06-23T01:06:17.885 INFO:teuthology.orchestra.run.smithi039.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 2.0 MB/s | 162 kB 00:00 2023-06-23T01:06:18.139 INFO:teuthology.orchestra.run.smithi073.stdout:(4/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 3.5 MB/s | 5.2 MB 00:01 2023-06-23T01:06:18.625 INFO:teuthology.orchestra.run.smithi073.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 3.2 MB/s | 1.6 MB 00:00 2023-06-23T01:06:18.641 INFO:teuthology.orchestra.run.smithi039.stdout:(4/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 3.0 MB/s | 2.3 MB 00:00 2023-06-23T01:06:18.713 INFO:teuthology.orchestra.run.smithi073.stdout:(6/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 1.9 MB/s | 2.3 MB 00:01 2023-06-23T01:06:19.125 INFO:teuthology.orchestra.run.smithi039.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 3.2 MB/s | 1.6 MB 00:00 2023-06-23T01:06:19.617 INFO:teuthology.orchestra.run.smithi073.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 4.0 MB/s | 3.9 MB 00:00 2023-06-23T01:06:20.218 INFO:teuthology.orchestra.run.smithi039.stdout:(6/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 3.6 MB/s | 3.9 MB 00:01 2023-06-23T01:06:22.061 INFO:teuthology.orchestra.run.smithi039.stdout:(7/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 4.4 MB/s | 23 MB 00:05 2023-06-23T01:06:23.171 INFO:teuthology.orchestra.run.smithi039.stdout:(8/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 829 kB/s | 5.2 MB 00:06 2023-06-23T01:06:23.236 INFO:teuthology.orchestra.run.smithi039.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 374 kB/s | 24 kB 00:00 2023-06-23T01:06:23.735 INFO:teuthology.orchestra.run.smithi073.stdout:(8/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8. 4.1 MB/s | 17 MB 00:04 2023-06-23T01:06:23.760 INFO:teuthology.orchestra.run.smithi073.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 990 kB/s | 24 kB 00:00 2023-06-23T01:06:23.784 INFO:teuthology.orchestra.run.smithi039.stdout:(10/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 4.7 MB/s | 17 MB 00:03 2023-06-23T01:06:23.865 INFO:teuthology.orchestra.run.smithi039.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 3.1 MB/s | 255 kB 00:00 2023-06-23T01:06:23.928 INFO:teuthology.orchestra.run.smithi039.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 395 kB/s | 25 kB 00:00 2023-06-23T01:06:24.131 INFO:teuthology.orchestra.run.smithi039.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 3.8 MB/s | 794 kB 00:00 2023-06-23T01:06:24.198 INFO:teuthology.orchestra.run.smithi039.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 2.5 MB/s | 174 kB 00:00 2023-06-23T01:06:24.261 INFO:teuthology.orchestra.run.smithi039.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 2.0 MB/s | 130 kB 00:00 2023-06-23T01:06:24.393 INFO:teuthology.orchestra.run.smithi039.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 3.8 MB/s | 506 kB 00:00 2023-06-23T01:06:24.624 INFO:teuthology.orchestra.run.smithi073.stdout:(10/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 2.8 MB/s | 17 MB 00:05 2023-06-23T01:06:24.730 INFO:teuthology.orchestra.run.smithi073.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 2.4 MB/s | 255 kB 00:00 2023-06-23T01:06:24.780 INFO:teuthology.orchestra.run.smithi073.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 493 kB/s | 25 kB 00:00 2023-06-23T01:06:24.989 INFO:teuthology.orchestra.run.smithi073.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 3.7 MB/s | 794 kB 00:00 2023-06-23T01:06:25.034 INFO:teuthology.orchestra.run.smithi073.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 3.8 MB/s | 174 kB 00:00 2023-06-23T01:06:25.080 INFO:teuthology.orchestra.run.smithi073.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 2.7 MB/s | 130 kB 00:00 2023-06-23T01:06:25.219 INFO:teuthology.orchestra.run.smithi073.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 3.6 MB/s | 506 kB 00:00 2023-06-23T01:06:25.458 INFO:teuthology.orchestra.run.smithi039.stdout:(17/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 4.9 MB/s | 17 MB 00:03 2023-06-23T01:06:25.477 INFO:teuthology.orchestra.run.smithi039.stdout:(18/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 5.1 MB/s | 5.5 MB 00:01 2023-06-23T01:06:25.507 INFO:teuthology.orchestra.run.smithi039.stdout:(19/120): python3-ceph-argparse-17.2.6-719.g42d 917 kB/s | 45 kB 00:00 2023-06-23T01:06:25.526 INFO:teuthology.orchestra.run.smithi039.stdout:(20/120): python3-ceph-common-17.2.6-719.g42ddc 2.3 MB/s | 115 kB 00:00 2023-06-23T01:06:25.564 INFO:teuthology.orchestra.run.smithi039.stdout:(21/120): python3-cephfs-17.2.6-719.g42ddc46e.e 3.3 MB/s | 190 kB 00:00 2023-06-23T01:06:25.644 INFO:teuthology.orchestra.run.smithi039.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 4.4 MB/s | 357 kB 00:00 2023-06-23T01:06:25.647 INFO:teuthology.orchestra.run.smithi039.stdout:(23/120): python3-rados-17.2.6-719.g42ddc46e.el 3.0 MB/s | 371 kB 00:00 2023-06-23T01:06:25.693 INFO:teuthology.orchestra.run.smithi039.stdout:(24/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 2.1 MB/s | 107 kB 00:00 2023-06-23T01:06:25.698 INFO:teuthology.orchestra.run.smithi039.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.7 MB/s | 91 kB 00:00 2023-06-23T01:06:25.764 INFO:teuthology.orchestra.run.smithi039.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 2.6 MB/s | 173 kB 00:00 2023-06-23T01:06:25.827 INFO:teuthology.orchestra.run.smithi039.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 382 kB/s | 23 kB 00:00 2023-06-23T01:06:25.885 INFO:teuthology.orchestra.run.smithi039.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 1.9 MB/s | 121 kB 00:00 2023-06-23T01:06:26.246 INFO:teuthology.orchestra.run.smithi039.stdout:(29/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 4.7 MB/s | 1.7 MB 00:00 2023-06-23T01:06:26.339 INFO:teuthology.orchestra.run.smithi073.stdout:(17/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 5.0 MB/s | 5.5 MB 00:01 2023-06-23T01:06:26.369 INFO:teuthology.orchestra.run.smithi039.stdout:(30/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 5.1 MB/s | 3.5 MB 00:00 2023-06-23T01:06:26.374 INFO:teuthology.orchestra.run.smithi073.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 1.2 MB/s | 45 kB 00:00 2023-06-23T01:06:26.411 INFO:teuthology.orchestra.run.smithi073.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 3.1 MB/s | 115 kB 00:00 2023-06-23T01:06:26.438 INFO:teuthology.orchestra.run.smithi039.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 3.4 MB/s | 238 kB 00:00 2023-06-23T01:06:26.455 INFO:teuthology.orchestra.run.smithi073.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 4.3 MB/s | 190 kB 00:00 2023-06-23T01:06:26.510 INFO:teuthology.orchestra.run.smithi039.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 656 kB/s | 47 kB 00:00 2023-06-23T01:06:26.528 INFO:teuthology.orchestra.run.smithi073.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 5.0 MB/s | 371 kB 00:00 2023-06-23T01:06:26.586 INFO:teuthology.orchestra.run.smithi039.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 192 kB/s | 14 kB 00:00 2023-06-23T01:06:26.602 INFO:teuthology.orchestra.run.smithi073.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 4.7 MB/s | 357 kB 00:00 2023-06-23T01:06:26.645 INFO:teuthology.orchestra.run.smithi073.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 2.5 MB/s | 107 kB 00:00 2023-06-23T01:06:26.665 INFO:teuthology.orchestra.run.smithi039.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 3.3 MB/s | 264 kB 00:00 2023-06-23T01:06:26.679 INFO:teuthology.orchestra.run.smithi073.stdout:(24/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 2.6 MB/s | 91 kB 00:00 2023-06-23T01:06:26.737 INFO:teuthology.orchestra.run.smithi039.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.1 MB/s | 80 kB 00:00 2023-06-23T01:06:26.799 INFO:teuthology.orchestra.run.smithi039.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 288 kB/s | 18 kB 00:00 2023-06-23T01:06:26.815 INFO:teuthology.orchestra.run.smithi039.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:26.817 INFO:teuthology.orchestra.run.smithi039.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:27.256 INFO:teuthology.orchestra.run.smithi039.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 670 kB/s | 306 kB 00:00 2023-06-23T01:06:27.257 INFO:teuthology.orchestra.run.smithi039.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:27.258 INFO:teuthology.orchestra.run.smithi039.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:27.306 INFO:teuthology.orchestra.run.smithi039.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 1.1 MB/s | 59 kB 00:00 2023-06-23T01:06:27.307 INFO:teuthology.orchestra.run.smithi039.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:27.308 INFO:teuthology.orchestra.run.smithi039.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:27.360 INFO:teuthology.orchestra.run.smithi073.stdout:(25/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 5.1 MB/s | 3.5 MB 00:00 2023-06-23T01:06:27.367 INFO:teuthology.orchestra.run.smithi039.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 1.2 MB/s | 75 kB 00:00 2023-06-23T01:06:27.404 INFO:teuthology.orchestra.run.smithi073.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 3.9 MB/s | 173 kB 00:00 2023-06-23T01:06:27.443 INFO:teuthology.orchestra.run.smithi073.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 586 kB/s | 23 kB 00:00 2023-06-23T01:06:27.480 INFO:teuthology.orchestra.run.smithi073.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 3.2 MB/s | 121 kB 00:00 2023-06-23T01:06:27.609 INFO:teuthology.orchestra.run.smithi039.stdout:(40/120): ceph-mgr-diskprediction-local-17.2.6- 5.4 MB/s | 7.4 MB 00:01 2023-06-23T01:06:27.656 INFO:teuthology.orchestra.run.smithi039.stdout:(41/120): python3-asyncssh-2.7.0-2.el8.noarch.r 1.6 MB/s | 476 kB 00:00 2023-06-23T01:06:27.752 INFO:teuthology.orchestra.run.smithi039.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 1.9 MB/s | 185 kB 00:00 2023-06-23T01:06:27.790 INFO:teuthology.orchestra.run.smithi073.stdout:(29/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 5.5 MB/s | 1.7 MB 00:00 2023-06-23T01:06:27.800 INFO:teuthology.orchestra.run.smithi039.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 695 kB/s | 33 kB 00:00 2023-06-23T01:06:27.824 INFO:teuthology.orchestra.run.smithi039.stdout:(44/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 203 kB/s | 44 kB 00:00 2023-06-23T01:06:27.847 INFO:teuthology.orchestra.run.smithi039.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 342 kB/s | 16 kB 00:00 2023-06-23T01:06:27.967 INFO:teuthology.orchestra.run.smithi039.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 1.2 MB/s | 173 kB 00:00 2023-06-23T01:06:28.019 INFO:teuthology.orchestra.run.smithi039.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 1.5 MB/s | 81 kB 00:00 2023-06-23T01:06:28.038 INFO:teuthology.orchestra.run.smithi039.stdout:(48/120): python3-cherrypy-18.4.0-1.el8.noarch. 2.0 MB/s | 384 kB 00:00 2023-06-23T01:06:28.066 INFO:teuthology.orchestra.run.smithi039.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 234 kB/s | 11 kB 00:00 2023-06-23T01:06:28.085 INFO:teuthology.orchestra.run.smithi039.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 371 kB/s | 18 kB 00:00 2023-06-23T01:06:28.134 INFO:teuthology.orchestra.run.smithi039.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 995 kB/s | 49 kB 00:00 2023-06-23T01:06:28.147 INFO:teuthology.orchestra.run.smithi073.stdout:(30/120): ceph-common-17.2.6-719.g42ddc46e.el8. 2.0 MB/s | 23 MB 00:11 2023-06-23T01:06:28.182 INFO:teuthology.orchestra.run.smithi039.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 1.2 MB/s | 59 kB 00:00 2023-06-23T01:06:28.230 INFO:teuthology.orchestra.run.smithi039.stdout:(53/120): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-06-23T01:06:28.259 INFO:teuthology.orchestra.run.smithi073.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 2.1 MB/s | 238 kB 00:00 2023-06-23T01:06:28.316 INFO:teuthology.orchestra.run.smithi073.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 830 kB/s | 47 kB 00:00 2023-06-23T01:06:28.372 INFO:teuthology.orchestra.run.smithi073.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 258 kB/s | 14 kB 00:00 2023-06-23T01:06:28.373 INFO:teuthology.orchestra.run.smithi039.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2023-06-23T01:06:28.420 INFO:teuthology.orchestra.run.smithi039.stdout:(55/120): python3-portend-2.6-1.el8.noarch.rpm 328 kB/s | 16 kB 00:00 2023-06-23T01:06:28.468 INFO:teuthology.orchestra.run.smithi039.stdout:(56/120): python3-repoze-lru-0.7-6.el8.noarch.r 708 kB/s | 34 kB 00:00 2023-06-23T01:06:28.490 INFO:teuthology.orchestra.run.smithi073.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 2.2 MB/s | 264 kB 00:00 2023-06-23T01:06:28.549 INFO:teuthology.orchestra.run.smithi073.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.3 MB/s | 80 kB 00:00 2023-06-23T01:06:28.578 INFO:teuthology.orchestra.run.smithi039.stdout:(57/120): python3-routes-2.4.1-12.el8.noarch.rp 1.7 MB/s | 196 kB 00:00 2023-06-23T01:06:28.607 INFO:teuthology.orchestra.run.smithi073.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 310 kB/s | 18 kB 00:00 2023-06-23T01:06:28.623 INFO:teuthology.orchestra.run.smithi073.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:28.626 INFO:teuthology.orchestra.run.smithi039.stdout:(58/120): python3-rsa-4.8-1.el8.noarch.rpm 1.3 MB/s | 61 kB 00:00 2023-06-23T01:06:28.626 INFO:teuthology.orchestra.run.smithi073.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:28.674 INFO:teuthology.orchestra.run.smithi039.stdout:(59/120): python3-simplegeneric-0.8.1-17.el8.no 394 kB/s | 19 kB 00:00 2023-06-23T01:06:28.690 INFO:teuthology.orchestra.run.smithi039.stdout:(60/120): python3-kubernetes-11.0.0-6.el8.noarc 1.9 MB/s | 1.2 MB 00:00 2023-06-23T01:06:28.721 INFO:teuthology.orchestra.run.smithi039.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 504 kB/s | 24 kB 00:00 2023-06-23T01:06:28.737 INFO:teuthology.orchestra.run.smithi039.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 615 kB/s | 29 kB 00:00 2023-06-23T01:06:28.773 INFO:teuthology.orchestra.run.smithi039.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 537 kB/s | 27 kB 00:00 2023-06-23T01:06:28.835 INFO:teuthology.orchestra.run.smithi073.stdout:(37/120): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-06-23T01:06:28.836 INFO:teuthology.orchestra.run.smithi073.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:28.837 INFO:teuthology.orchestra.run.smithi073.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:28.872 INFO:teuthology.orchestra.run.smithi073.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 1.6 MB/s | 59 kB 00:00 2023-06-23T01:06:28.872 INFO:teuthology.orchestra.run.smithi073.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:28.873 INFO:teuthology.orchestra.run.smithi073.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:06:28.879 INFO:teuthology.orchestra.run.smithi039.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 1.8 MB/s | 255 kB 00:00 2023-06-23T01:06:28.909 INFO:teuthology.orchestra.run.smithi073.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 2.0 MB/s | 75 kB 00:00 2023-06-23T01:06:28.921 INFO:teuthology.orchestra.run.smithi039.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 1.7 MB/s | 251 kB 00:00 2023-06-23T01:06:28.927 INFO:teuthology.orchestra.run.smithi039.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 1.3 MB/s | 61 kB 00:00 2023-06-23T01:06:28.970 INFO:teuthology.orchestra.run.smithi039.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 1.7 MB/s | 85 kB 00:00 2023-06-23T01:06:28.975 INFO:teuthology.orchestra.run.smithi039.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 479 kB/s | 23 kB 00:00 2023-06-23T01:06:28.980 INFO:teuthology.orchestra.run.smithi039.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-06-23T01:06:28.982 INFO:teuthology.orchestra.run.smithi073.stdout:(40/120): python3-asyncssh-2.7.0-2.el8.noarch.r 6.4 MB/s | 476 kB 00:00 2023-06-23T01:06:29.013 INFO:teuthology.orchestra.run.smithi073.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.4 MB/s | 44 kB 00:00 2023-06-23T01:06:29.050 INFO:teuthology.orchestra.run.smithi073.stdout:(42/120): python3-beautifulsoup4-4.6.3-2.el8.1. 5.0 MB/s | 185 kB 00:00 2023-06-23T01:06:29.082 INFO:teuthology.orchestra.run.smithi073.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 1.0 MB/s | 33 kB 00:00 2023-06-23T01:06:29.125 INFO:teuthology.orchestra.run.smithi073.stdout:(44/120): ceph-mgr-diskprediction-local-17.2.6- 5.5 MB/s | 7.4 MB 00:01 2023-06-23T01:06:29.125 INFO:teuthology.orchestra.run.smithi073.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 376 kB/s | 16 kB 00:00 2023-06-23T01:06:29.181 INFO:teuthology.orchestra.run.smithi073.stdout:(46/120): python3-cherrypy-18.4.0-1.el8.noarch. 6.8 MB/s | 384 kB 00:00 2023-06-23T01:06:29.223 INFO:teuthology.orchestra.run.smithi073.stdout:(47/120): python3-google-auth-1.1.1-10.el8.noar 1.9 MB/s | 81 kB 00:00 2023-06-23T01:06:29.257 INFO:teuthology.orchestra.run.smithi039.stdout:(70/120): python3-scipy-1.0.0-20.module+el8.1.0 53 MB/s | 15 MB 00:00 2023-06-23T01:06:29.271 INFO:teuthology.orchestra.run.smithi039.stdout:(71/120): python3-prettytable-0.7.2-14.el8.noar 3.2 MB/s | 44 kB 00:00 2023-06-23T01:06:29.282 INFO:teuthology.orchestra.run.smithi073.stdout:(48/120): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2023-06-23T01:06:29.283 INFO:teuthology.orchestra.run.smithi039.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 2.1 MB/s | 27 kB 00:00 2023-06-23T01:06:29.297 INFO:teuthology.orchestra.run.smithi039.stdout:(73/120): jq-1.5-12.el8.x86_64.rpm 11 MB/s | 161 kB 00:00 2023-06-23T01:06:29.313 INFO:teuthology.orchestra.run.smithi039.stdout:(74/120): flex-2.6.1-9.el8.x86_64.rpm 20 MB/s | 320 kB 00:00 2023-06-23T01:06:29.331 INFO:teuthology.orchestra.run.smithi039.stdout:(75/120): python3-werkzeug-0.12.2-4.el8.noarch. 25 MB/s | 457 kB 00:00 2023-06-23T01:06:29.344 INFO:teuthology.orchestra.run.smithi073.stdout:(49/120): python3-jaraco-functools-2.0-4.el8.no 283 kB/s | 18 kB 00:00 2023-06-23T01:06:29.404 INFO:teuthology.orchestra.run.smithi073.stdout:(50/120): python3-cheroot-8.5.2-1.el8.noarch.rp 620 kB/s | 173 kB 00:00 2023-06-23T01:06:29.404 INFO:teuthology.orchestra.run.smithi039.stdout:(76/120): python3-babel-2.5.1-5.el8.noarch.rpm 66 MB/s | 4.8 MB 00:00 2023-06-23T01:06:29.416 INFO:teuthology.orchestra.run.smithi073.stdout:(51/120): python3-kubernetes-11.0.0-6.el8.noarc 16 MB/s | 1.2 MB 00:00 2023-06-23T01:06:29.417 INFO:teuthology.orchestra.run.smithi039.stdout:(77/120): python3-markupsafe-0.23-19.el8.x86_64 2.9 MB/s | 39 kB 00:00 2023-06-23T01:06:29.438 INFO:teuthology.orchestra.run.smithi039.stdout:(78/120): bison-3.0.4-10.el8.x86_64.rpm 33 MB/s | 688 kB 00:00 2023-06-23T01:06:29.444 INFO:teuthology.orchestra.run.smithi073.stdout:(52/120): python3-logutils-0.3.5-11.el8.noarch. 1.2 MB/s | 49 kB 00:00 2023-06-23T01:06:29.448 INFO:teuthology.orchestra.run.smithi073.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 1.9 MB/s | 59 kB 00:00 2023-06-23T01:06:29.454 INFO:teuthology.orchestra.run.smithi039.stdout:(79/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 16 MB/s | 259 kB 00:00 2023-06-23T01:06:29.469 INFO:teuthology.orchestra.run.smithi039.stdout:(80/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 15 MB/s | 225 kB 00:00 2023-06-23T01:06:29.482 INFO:teuthology.orchestra.run.smithi073.stdout:(54/120): python3-pecan-1.3.2-9.el8.noarch.rpm 8.0 MB/s | 283 kB 00:00 2023-06-23T01:06:29.483 INFO:teuthology.orchestra.run.smithi039.stdout:(81/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 9.1 MB/s | 126 kB 00:00 2023-06-23T01:06:29.485 INFO:teuthology.orchestra.run.smithi073.stdout:(55/120): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2023-06-23T01:06:29.516 INFO:teuthology.orchestra.run.smithi073.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 462 kB/s | 16 kB 00:00 2023-06-23T01:06:29.526 INFO:teuthology.orchestra.run.smithi073.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 836 kB/s | 34 kB 00:00 2023-06-23T01:06:29.542 INFO:teuthology.orchestra.run.smithi039.stdout:(82/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 64 MB/s | 3.7 MB 00:00 2023-06-23T01:06:29.550 INFO:teuthology.orchestra.run.smithi073.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 5.7 MB/s | 196 kB 00:00 2023-06-23T01:06:29.558 INFO:teuthology.orchestra.run.smithi039.stdout:(83/120): oniguruma-6.8.2-2.el8.x86_64.rpm 12 MB/s | 187 kB 00:00 2023-06-23T01:06:29.568 INFO:teuthology.orchestra.run.smithi073.stdout:(59/120): python3-rsa-4.8-1.el8.noarch.rpm 1.4 MB/s | 61 kB 00:00 2023-06-23T01:06:29.576 INFO:teuthology.orchestra.run.smithi039.stdout:(84/120): python3-jinja2-2.10.1-2.el8_0.noarch. 29 MB/s | 538 kB 00:00 2023-06-23T01:06:29.580 INFO:teuthology.orchestra.run.smithi073.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 615 kB/s | 19 kB 00:00 2023-06-23T01:06:29.592 INFO:teuthology.orchestra.run.smithi039.stdout:(85/120): socat-1.7.3.3-2.el8.x86_64.rpm 19 MB/s | 302 kB 00:00 2023-06-23T01:06:29.604 INFO:teuthology.orchestra.run.smithi073.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 667 kB/s | 24 kB 00:00 2023-06-23T01:06:29.606 INFO:teuthology.orchestra.run.smithi039.stdout:(86/120): python3-mako-1.0.6-13.el8.noarch.rpm 11 MB/s | 157 kB 00:00 2023-06-23T01:06:29.617 INFO:teuthology.orchestra.run.smithi073.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 813 kB/s | 29 kB 00:00 2023-06-23T01:06:29.619 INFO:teuthology.orchestra.run.smithi039.stdout:(87/120): python3-pyasn1-modules-0.3.7-6.el8.no 8.2 MB/s | 110 kB 00:00 2023-06-23T01:06:29.635 INFO:teuthology.orchestra.run.smithi039.stdout:(88/120): librdkafka-0.11.4-1.el8.x86_64.rpm 22 MB/s | 353 kB 00:00 2023-06-23T01:06:29.640 INFO:teuthology.orchestra.run.smithi073.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 784 kB/s | 27 kB 00:00 2023-06-23T01:06:29.647 INFO:teuthology.orchestra.run.smithi039.stdout:(89/120): python3-jsonpointer-1.10-11.el8.noarc 1.6 MB/s | 20 kB 00:00 2023-06-23T01:06:29.651 INFO:teuthology.orchestra.run.smithi073.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 7.4 MB/s | 255 kB 00:00 2023-06-23T01:06:29.668 INFO:teuthology.orchestra.run.smithi039.stdout:(90/120): libtool-2.4.6-25.el8.x86_64.rpm 34 MB/s | 709 kB 00:00 2023-06-23T01:06:29.681 INFO:teuthology.orchestra.run.smithi039.stdout:(91/120): python3-pytz-2017.2-9.el8.noarch.rpm 4.2 MB/s | 54 kB 00:00 2023-06-23T01:06:29.682 INFO:teuthology.orchestra.run.smithi073.stdout:(65/120): python3-websocket-client-0.56.0-5.el8 1.9 MB/s | 61 kB 00:00 2023-06-23T01:06:29.722 INFO:teuthology.orchestra.run.smithi073.stdout:(66/120): python3-webtest-2.0.33-1.el8.noarch.r 2.1 MB/s | 85 kB 00:00 2023-06-23T01:06:29.723 INFO:teuthology.orchestra.run.smithi073.stdout:(67/120): python3-webob-1.8.5-1.el8.1.noarch.rp 2.9 MB/s | 251 kB 00:00 2023-06-23T01:06:29.743 INFO:teuthology.orchestra.run.smithi039.stdout:(92/120): thrift-0.13.0-2.el8.x86_64.rpm 2.2 MB/s | 1.7 MB 00:00 2023-06-23T01:06:29.753 INFO:teuthology.orchestra.run.smithi073.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 750 kB/s | 23 kB 00:00 2023-06-23T01:06:29.757 INFO:teuthology.orchestra.run.smithi073.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 16 MB/s | 70 kB 00:00 2023-06-23T01:06:29.758 INFO:teuthology.orchestra.run.smithi039.stdout:(93/120): openblas-threads-0.3.12-1.el8.x86_64. 62 MB/s | 4.7 MB 00:00 2023-06-23T01:06:29.790 INFO:teuthology.orchestra.run.smithi073.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 578 kB/s | 19 kB 00:00 2023-06-23T01:06:29.791 INFO:teuthology.orchestra.run.smithi039.stdout:(94/120): automake-1.16.1-7.el8.noarch.rpm 28 MB/s | 713 kB 00:00 2023-06-23T01:06:29.806 INFO:teuthology.orchestra.run.smithi073.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 2.7 MB/s | 43 kB 00:00 2023-06-23T01:06:29.806 INFO:teuthology.orchestra.run.smithi039.stdout:(95/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 1.2 MB/s | 19 kB 00:00 2023-06-23T01:06:29.821 INFO:teuthology.orchestra.run.smithi039.stdout:(96/120): python3-jwt-1.6.1-2.el8.noarch.rpm 2.9 MB/s | 43 kB 00:00 2023-06-23T01:06:29.822 INFO:teuthology.orchestra.run.smithi073.stdout:(72/120): python3-oauthlib-2.1.0-1.el8.noarch.r 9.3 MB/s | 155 kB 00:00 2023-06-23T01:06:29.840 INFO:teuthology.orchestra.run.smithi073.stdout:(73/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 12 MB/s | 220 kB 00:00 2023-06-23T01:06:29.845 INFO:teuthology.orchestra.run.smithi039.stdout:(97/120): openblas-0.3.12-1.el8.x86_64.rpm 45 MB/s | 4.6 MB 00:00 2023-06-23T01:06:29.846 INFO:teuthology.orchestra.run.smithi073.stdout:(74/120): thrift-0.13.0-2.el8.x86_64.rpm 14 MB/s | 1.7 MB 00:00 2023-06-23T01:06:29.848 INFO:teuthology.orchestra.run.smithi039.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 5.7 MB/s | 155 kB 00:00 2023-06-23T01:06:29.854 INFO:teuthology.orchestra.run.smithi073.stdout:(75/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.3 MB/s | 48 kB 00:00 2023-06-23T01:06:29.863 INFO:teuthology.orchestra.run.smithi039.stdout:(99/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 3.2 MB/s | 48 kB 00:00 2023-06-23T01:06:29.864 INFO:teuthology.orchestra.run.smithi039.stdout:(100/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 12 MB/s | 220 kB 00:00 2023-06-23T01:06:29.872 INFO:teuthology.orchestra.run.smithi073.stdout:(76/120): python3-requests-oauthlib-1.0.0-1.el8 2.4 MB/s | 43 kB 00:00 2023-06-23T01:06:29.878 INFO:teuthology.orchestra.run.smithi039.stdout:(101/120): python3-requests-oauthlib-1.0.0-1.el 2.9 MB/s | 43 kB 00:00 2023-06-23T01:06:29.882 INFO:teuthology.orchestra.run.smithi039.stdout:(102/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 12 MB/s | 229 kB 00:00 2023-06-23T01:06:29.885 INFO:teuthology.orchestra.run.smithi073.stdout:(77/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.7 MB/s | 229 kB 00:00 2023-06-23T01:06:29.900 INFO:teuthology.orchestra.run.smithi039.stdout:(103/120): libverto-devel-0.3.0-5.el8.x86_64.rp 1.0 MB/s | 18 kB 00:00 2023-06-23T01:06:29.908 INFO:teuthology.orchestra.run.smithi073.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 808 kB/s | 18 kB 00:00 2023-06-23T01:06:29.917 INFO:teuthology.orchestra.run.smithi073.stdout:(79/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 52 MB/s | 2.3 MB 00:00 2023-06-23T01:06:29.922 INFO:teuthology.orchestra.run.smithi039.stdout:(104/120): openssl-devel-1.1.1g-15.el8_3.x86_64 54 MB/s | 2.3 MB 00:00 2023-06-23T01:06:29.926 INFO:teuthology.orchestra.run.smithi073.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 33 MB/s | 605 kB 00:00 2023-06-23T01:06:29.929 INFO:teuthology.orchestra.run.smithi039.stdout:(105/120): pcre2-devel-10.32-2.el8.x86_64.rpm 20 MB/s | 605 kB 00:00 2023-06-23T01:06:29.933 INFO:teuthology.orchestra.run.smithi073.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 5.0 MB/s | 83 kB 00:00 2023-06-23T01:06:29.939 INFO:teuthology.orchestra.run.smithi039.stdout:(106/120): fuse-2.9.7-12.el8.x86_64.rpm 4.9 MB/s | 83 kB 00:00 2023-06-23T01:06:29.941 INFO:teuthology.orchestra.run.smithi073.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 3.8 MB/s | 59 kB 00:00 2023-06-23T01:06:29.950 INFO:teuthology.orchestra.run.smithi073.stdout:(83/120): libquadmath-8.4.1-1.el8.x86_64.rpm 10 MB/s | 169 kB 00:00 2023-06-23T01:06:29.952 INFO:teuthology.orchestra.run.smithi039.stdout:(107/120): elfutils-libelf-devel-0.182-3.el8.x8 2.6 MB/s | 59 kB 00:00 2023-06-23T01:06:29.959 INFO:teuthology.orchestra.run.smithi073.stdout:(84/120): libcom_err-devel-1.45.6-1.el8.x86_64. 2.2 MB/s | 38 kB 00:00 2023-06-23T01:06:29.960 INFO:teuthology.orchestra.run.smithi039.stdout:(108/120): libquadmath-8.4.1-1.el8.x86_64.rpm 7.7 MB/s | 169 kB 00:00 2023-06-23T01:06:29.966 INFO:teuthology.orchestra.run.smithi073.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.3 MB/s | 86 kB 00:00 2023-06-23T01:06:29.969 INFO:teuthology.orchestra.run.smithi039.stdout:(109/120): libcom_err-devel-1.45.6-1.el8.x86_64 2.2 MB/s | 38 kB 00:00 2023-06-23T01:06:29.979 INFO:teuthology.orchestra.run.smithi073.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.8 MB/s | 58 kB 00:00 2023-06-23T01:06:29.985 INFO:teuthology.orchestra.run.smithi073.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.5 MB/s | 47 kB 00:00 2023-06-23T01:06:29.989 INFO:teuthology.orchestra.run.smithi039.stdout:(110/120): libsepol-devel-2.9-2.el8.x86_64.rpm 3.0 MB/s | 86 kB 00:00 2023-06-23T01:06:29.993 INFO:teuthology.orchestra.run.smithi039.stdout:(111/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 2.3 MB/s | 58 kB 00:00 2023-06-23T01:06:30.002 INFO:teuthology.orchestra.run.smithi073.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 24 MB/s | 559 kB 00:00 2023-06-23T01:06:30.005 INFO:teuthology.orchestra.run.smithi039.stdout:(112/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 2.8 MB/s | 47 kB 00:00 2023-06-23T01:06:30.009 INFO:teuthology.orchestra.run.smithi073.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2023-06-23T01:06:30.015 INFO:teuthology.orchestra.run.smithi039.stdout:(113/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 25 MB/s | 559 kB 00:00 2023-06-23T01:06:30.018 INFO:teuthology.orchestra.run.smithi073.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 4.1 MB/s | 70 kB 00:00 2023-06-23T01:06:30.028 INFO:teuthology.orchestra.run.smithi073.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 10 MB/s | 200 kB 00:00 2023-06-23T01:06:30.029 INFO:teuthology.orchestra.run.smithi039.stdout:(114/120): libgfortran-8.4.1-1.el8.x86_64.rpm 26 MB/s | 639 kB 00:00 2023-06-23T01:06:30.031 INFO:teuthology.orchestra.run.smithi039.stdout:(115/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 4.5 MB/s | 70 kB 00:00 2023-06-23T01:06:30.043 INFO:teuthology.orchestra.run.smithi073.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 4.0 MB/s | 97 kB 00:00 2023-06-23T01:06:30.065 INFO:teuthology.orchestra.run.smithi039.stdout:(116/120): libselinux-devel-2.9-5.el8.x86_64.rp 5.5 MB/s | 200 kB 00:00 2023-06-23T01:06:30.071 INFO:teuthology.orchestra.run.smithi039.stdout:(117/120): libuuid-devel-2.32.1-27.el8.x86_64.r 2.4 MB/s | 97 kB 00:00 2023-06-23T01:06:30.079 INFO:teuthology.orchestra.run.smithi073.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 1.2 MB/s | 44 kB 00:00 2023-06-23T01:06:30.095 INFO:teuthology.orchestra.run.smithi073.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.7 MB/s | 27 kB 00:00 2023-06-23T01:06:30.112 INFO:teuthology.orchestra.run.smithi073.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 9.5 MB/s | 161 kB 00:00 2023-06-23T01:06:30.131 INFO:teuthology.orchestra.run.smithi073.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 17 MB/s | 320 kB 00:00 2023-06-23T01:06:30.151 INFO:teuthology.orchestra.run.smithi073.stdout:(97/120): python3-werkzeug-0.12.2-4.el8.noarch. 22 MB/s | 457 kB 00:00 2023-06-23T01:06:30.269 INFO:teuthology.orchestra.run.smithi073.stdout:(98/120): python3-scipy-1.0.0-20.module+el8.1.0 60 MB/s | 15 MB 00:00 2023-06-23T01:06:30.285 INFO:teuthology.orchestra.run.smithi073.stdout:(99/120): python3-markupsafe-0.23-19.el8.x86_64 2.4 MB/s | 39 kB 00:00 2023-06-23T01:06:30.310 INFO:teuthology.orchestra.run.smithi073.stdout:(100/120): bison-3.0.4-10.el8.x86_64.rpm 28 MB/s | 688 kB 00:00 2023-06-23T01:06:30.325 INFO:teuthology.orchestra.run.smithi073.stdout:(101/120): python3-babel-2.5.1-5.el8.noarch.rpm 28 MB/s | 4.8 MB 00:00 2023-06-23T01:06:30.328 INFO:teuthology.orchestra.run.smithi073.stdout:(102/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 14 MB/s | 259 kB 00:00 2023-06-23T01:06:30.340 INFO:teuthology.orchestra.run.smithi073.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 15 MB/s | 225 kB 00:00 2023-06-23T01:06:30.348 INFO:teuthology.orchestra.run.smithi073.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 6.3 MB/s | 126 kB 00:00 2023-06-23T01:06:30.366 INFO:teuthology.orchestra.run.smithi073.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 11 MB/s | 187 kB 00:00 2023-06-23T01:06:30.386 INFO:teuthology.orchestra.run.smithi073.stdout:(106/120): python3-jinja2-2.10.1-2.el8_0.noarch 26 MB/s | 538 kB 00:00 2023-06-23T01:06:30.403 INFO:teuthology.orchestra.run.smithi073.stdout:(107/120): python3-numpy-1.14.3-9.el8.x86_64.rp 60 MB/s | 3.7 MB 00:00 2023-06-23T01:06:30.406 INFO:teuthology.orchestra.run.smithi073.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 15 MB/s | 302 kB 00:00 2023-06-23T01:06:30.417 INFO:teuthology.orchestra.run.smithi073.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 11 MB/s | 157 kB 00:00 2023-06-23T01:06:30.420 INFO:teuthology.orchestra.run.smithi073.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 7.6 MB/s | 110 kB 00:00 2023-06-23T01:06:30.434 INFO:teuthology.orchestra.run.smithi073.stdout:(111/120): python3-jsonpointer-1.10-11.el8.noar 1.5 MB/s | 20 kB 00:00 2023-06-23T01:06:30.434 INFO:teuthology.orchestra.run.smithi073.stdout:(112/120): librdkafka-0.11.4-1.el8.x86_64.rpm 19 MB/s | 353 kB 00:00 2023-06-23T01:06:30.452 INFO:teuthology.orchestra.run.smithi073.stdout:(113/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.1 MB/s | 54 kB 00:00 2023-06-23T01:06:30.454 INFO:teuthology.orchestra.run.smithi073.stdout:(114/120): libtool-2.4.6-25.el8.x86_64.rpm 33 MB/s | 709 kB 00:00 2023-06-23T01:06:30.530 INFO:teuthology.orchestra.run.smithi073.stdout:(115/120): openblas-0.3.12-1.el8.x86_64.rpm 61 MB/s | 4.6 MB 00:00 2023-06-23T01:06:30.545 INFO:teuthology.orchestra.run.smithi073.stdout:(116/120): openblas-threads-0.3.12-1.el8.x86_64 51 MB/s | 4.7 MB 00:00 2023-06-23T01:06:30.553 INFO:teuthology.orchestra.run.smithi073.stdout:(117/120): automake-1.16.1-7.el8.noarch.rpm 30 MB/s | 713 kB 00:00 2023-06-23T01:06:31.194 INFO:teuthology.orchestra.run.smithi039.stdout:(118/120): librados2-17.2.6-719.g42ddc46e.el8.x 3.1 MB/s | 3.5 MB 00:01 2023-06-23T01:06:31.206 INFO:teuthology.orchestra.run.smithi039.stdout:(119/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 3.2 MB/s | 3.6 MB 00:01 2023-06-23T01:06:31.407 INFO:teuthology.orchestra.run.smithi073.stdout:(118/120): ceph-test-17.2.6-719.g42ddc46e.el8.x 6.8 MB/s | 52 MB 00:07 2023-06-23T01:06:31.452 INFO:teuthology.orchestra.run.smithi073.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 3.8 MB/s | 3.5 MB 00:00 2023-06-23T01:06:31.808 INFO:teuthology.orchestra.run.smithi073.stdout:(120/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 2.9 MB/s | 3.6 MB 00:01 2023-06-23T01:06:31.811 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:06:31.812 INFO:teuthology.orchestra.run.smithi073.stdout:Total 13 MB/s | 201 MB 00:15 2023-06-23T01:06:31.860 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2023-06-23T01:06:31.898 INFO:teuthology.orchestra.run.smithi039.stdout:(120/120): ceph-test-17.2.6-719.g42ddc46e.el8.x 6.0 MB/s | 52 MB 00:08 2023-06-23T01:06:31.905 INFO:teuthology.orchestra.run.smithi039.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:06:31.906 INFO:teuthology.orchestra.run.smithi039.stdout:Total 13 MB/s | 201 MB 00:15 2023-06-23T01:06:31.965 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction check 2023-06-23T01:06:31.972 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2023-06-23T01:06:31.973 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2023-06-23T01:06:32.094 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction check succeeded. 2023-06-23T01:06:32.094 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction test 2023-06-23T01:06:33.660 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2023-06-23T01:06:33.664 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2023-06-23T01:06:33.807 INFO:teuthology.orchestra.run.smithi039.stdout:Transaction test succeeded. 2023-06-23T01:06:33.811 INFO:teuthology.orchestra.run.smithi039.stdout:Running transaction 2023-06-23T01:06:38.719 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2023-06-23T01:06:38.913 INFO:teuthology.orchestra.run.smithi039.stdout: Preparing : 1/1 2023-06-23T01:06:38.920 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T01:06:39.084 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T01:06:39.106 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T01:06:39.268 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-06-23T01:06:39.269 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T01:06:39.448 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T01:06:39.472 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T01:06:39.579 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T01:06:39.772 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T01:06:39.859 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T01:06:40.056 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:06:40.182 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T01:06:40.456 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:06:40.514 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:06:40.672 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:06:40.938 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:06:41.048 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:06:41.106 INFO:teuthology.orchestra.run.smithi039.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:06:41.235 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:06:41.364 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:06:41.472 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:06:41.653 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:06:41.714 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:06:41.806 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:06:42.006 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-06-23T01:06:42.148 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:06:42.656 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T01:06:42.829 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T01:06:42.830 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:06:42.946 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:06:42.981 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:06:43.098 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:06:43.192 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T01:06:43.329 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T01:06:43.351 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T01:06:43.505 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T01:06:43.513 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:06:43.621 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:06:43.697 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:06:43.772 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:06:43.807 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:06:43.849 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:06:43.939 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:06:44.000 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:06:44.001 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-06-23T01:06:44.147 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-06-23T01:06:44.245 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-06-23T01:06:44.280 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T01:06:44.417 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T01:06:44.479 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-06-23T01:06:44.572 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T01:06:44.622 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-06-23T01:06:44.772 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T01:06:44.912 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T01:06:45.080 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T01:06:45.157 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T01:06:45.305 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T01:06:45.634 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T01:06:45.780 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T01:06:45.870 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T01:06:46.038 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T01:06:46.309 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T01:06:46.812 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T01:06:46.938 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T01:06:47.495 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-06-23T01:06:47.656 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-06-23T01:06:47.672 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T01:06:47.812 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-06-23T01:06:47.925 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-06-23T01:06:48.057 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-06-23T01:06:48.174 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-06-23T01:06:48.321 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-06-23T01:06:48.400 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T01:06:48.538 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T01:06:48.544 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T01:06:48.665 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T01:06:48.695 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T01:06:48.816 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T01:06:48.955 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:06:49.058 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:06:49.169 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-06-23T01:06:49.412 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-06-23T01:06:49.621 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T01:06:50.126 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-06-23T01:06:50.446 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-06-23T01:06:50.701 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-06-23T01:06:50.954 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-06-23T01:06:51.201 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-06-23T01:06:51.462 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T01:06:51.689 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T01:06:52.036 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-06-23T01:06:53.024 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-06-23T01:06:53.400 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-06-23T01:06:53.634 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T01:06:53.649 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-06-23T01:06:53.909 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T01:06:53.926 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-06-23T01:06:54.090 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T01:06:54.137 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T01:06:54.297 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T01:06:54.307 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T01:06:54.504 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T01:06:54.505 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T01:06:54.714 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T01:06:54.776 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T01:06:54.900 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T01:06:54.977 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-06-23T01:06:55.062 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-06-23T01:06:55.169 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-06-23T01:06:55.197 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-06-23T01:06:55.341 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-06-23T01:06:55.367 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-06-23T01:06:55.523 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-06-23T01:06:55.523 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-06-23T01:06:55.679 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T01:06:55.759 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T01:06:55.767 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-06-23T01:06:55.937 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-06-23T01:06:55.938 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-06-23T01:06:56.099 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-06-23T01:06:56.100 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-06-23T01:06:56.303 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-06-23T01:06:56.392 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-06-23T01:06:56.517 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-06-23T01:06:56.620 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-06-23T01:06:56.689 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-06-23T01:06:56.831 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-06-23T01:06:56.835 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-06-23T01:06:56.974 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-06-23T01:06:57.160 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-06-23T01:06:57.178 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-06-23T01:06:57.340 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-06-23T01:06:57.361 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-06-23T01:06:57.519 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-06-23T01:06:57.626 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-06-23T01:06:57.710 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-06-23T01:06:57.909 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-06-23T01:06:57.925 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-06-23T01:06:58.078 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T01:06:58.143 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T01:06:58.154 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-06-23T01:06:58.312 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-06-23T01:06:58.432 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-06-23T01:06:58.465 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-06-23T01:06:58.562 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T01:06:58.604 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T01:06:58.732 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T01:06:58.752 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T01:06:58.974 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T01:06:58.989 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T01:06:59.212 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T01:06:59.304 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T01:06:59.393 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T01:06:59.477 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T01:06:59.545 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T01:06:59.650 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T01:06:59.723 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T01:06:59.798 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T01:06:59.903 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T01:06:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T01:07:00.106 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T01:07:00.191 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T01:07:00.338 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T01:07:00.390 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T01:07:00.514 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T01:07:00.548 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T01:07:00.642 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T01:07:00.676 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T01:07:00.879 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T01:07:00.907 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T01:07:01.097 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T01:07:01.099 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T01:07:01.317 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T01:07:01.335 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T01:07:01.582 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T01:07:01.628 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T01:07:01.860 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T01:07:01.861 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T01:07:02.036 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T01:07:02.060 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T01:07:02.266 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T01:07:02.368 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T01:07:03.433 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T01:07:03.465 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T01:07:03.727 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T01:07:03.750 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T01:07:03.922 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T01:07:03.987 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T01:07:04.246 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T01:07:04.345 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T01:07:04.476 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T01:07:04.599 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T01:07:04.815 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T01:07:04.949 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T01:07:05.024 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T01:07:05.187 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T01:07:05.204 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T01:07:05.369 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T01:07:05.411 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T01:07:05.569 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T01:07:10.526 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:07:10.596 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:07:10.727 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:07:10.802 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:07:11.306 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:07:11.441 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:07:11.570 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:07:11.653 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:07:11.695 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:07:11.789 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:07:11.818 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:07:11.918 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:07:29.424 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:07:29.425 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /sys 2023-06-23T01:07:29.425 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /proc 2023-06-23T01:07:29.425 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /dev 2023-06-23T01:07:29.425 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /run 2023-06-23T01:07:29.425 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /mnt 2023-06-23T01:07:29.425 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /var/tmp 2023-06-23T01:07:29.426 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /home 2023-06-23T01:07:29.426 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /tmp 2023-06-23T01:07:29.426 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /dev 2023-06-23T01:07:29.426 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:07:29.807 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:07:30.131 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:07:30.131 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /sys 2023-06-23T01:07:30.132 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /proc 2023-06-23T01:07:30.132 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-06-23T01:07:30.132 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /run 2023-06-23T01:07:30.132 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /mnt 2023-06-23T01:07:30.132 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /var/tmp 2023-06-23T01:07:30.132 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /home 2023-06-23T01:07:30.132 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /tmp 2023-06-23T01:07:30.133 INFO:teuthology.orchestra.run.smithi039.stdout:skipping the directory /dev 2023-06-23T01:07:30.133 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:30.490 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:07:31.295 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:07:31.442 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:07:31.665 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:07:31.823 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:07:32.064 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:07:32.214 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:07:32.216 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:07:32.398 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:07:32.415 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:07:32.451 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:07:32.582 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:07:32.995 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:07:33.132 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:07:33.150 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:07:35.141 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:07:35.280 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:07:35.455 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T01:07:35.661 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T01:07:35.839 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:07:36.021 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:07:36.054 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:07:36.177 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:07:36.274 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:07:36.354 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T01:07:36.530 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T01:07:36.697 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:07:36.923 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:07:37.157 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:07:37.242 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:07:37.405 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:07:37.504 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:07:37.864 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T01:07:38.033 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:07:38.070 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:07:38.188 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:07:38.222 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:07:38.233 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:07:38.348 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:07:38.449 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:07:38.805 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T01:07:38.974 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:07:39.012 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:07:39.164 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:07:39.176 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:07:39.811 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T01:07:40.058 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:07:40.134 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:07:40.272 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:07:40.315 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:07:40.455 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:07:40.805 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:07:40.868 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T01:07:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:07:41.065 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:07:41.146 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:07:41.272 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:07:41.317 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:07:41.464 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:07:41.834 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:07:42.022 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:07:45.894 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:07:46.830 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T01:07:47.202 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-06-23T01:07:47.271 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:07:47.368 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 110/122 2023-06-23T01:07:47.573 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T01:07:47.586 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-06-23T01:07:47.737 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T01:07:47.770 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T01:07:47.782 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T01:07:47.933 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T01:07:48.066 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T01:07:48.237 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T01:07:48.442 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T01:07:48.480 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T01:07:48.687 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T01:07:48.821 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-06-23T01:07:48.870 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T01:07:48.932 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T01:07:49.001 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 111/122 2023-06-23T01:07:49.120 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T01:07:49.129 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T01:07:49.149 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-06-23T01:07:49.247 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-06-23T01:07:49.248 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:07:49.284 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T01:07:49.418 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T01:07:49.440 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:07:49.440 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:07:49.563 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T01:07:49.716 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T01:07:49.855 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T01:07:49.987 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:07:49.987 INFO:teuthology.orchestra.run.smithi073.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T01:07:49.987 INFO:teuthology.orchestra.run.smithi073.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T01:07:49.987 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:07:50.071 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-06-23T01:07:50.279 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T01:07:50.332 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T01:07:50.333 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T01:07:50.333 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T01:07:50.333 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:07:50.333 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T01:07:50.333 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:07:50.334 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:07:50.334 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:07:50.334 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T01:07:50.334 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:07:50.334 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T01:07:50.334 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T01:07:50.334 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:07:50.334 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T01:07:50.335 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T01:07:50.335 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T01:07:50.335 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T01:07:50.335 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T01:07:50.335 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T01:07:50.335 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T01:07:50.335 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T01:07:50.335 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T01:07:50.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T01:07:50.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T01:07:50.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T01:07:50.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T01:07:50.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T01:07:50.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:07:50.336 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T01:07:50.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T01:07:50.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T01:07:50.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T01:07:50.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T01:07:50.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T01:07:50.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T01:07:50.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T01:07:50.337 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T01:07:50.338 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T01:07:50.338 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T01:07:50.338 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T01:07:50.338 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T01:07:50.338 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T01:07:50.338 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T01:07:50.338 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T01:07:50.339 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T01:07:50.339 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T01:07:50.339 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T01:07:50.339 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T01:07:50.339 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T01:07:50.339 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T01:07:50.339 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T01:07:50.339 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T01:07:50.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T01:07:50.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T01:07:50.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T01:07:50.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T01:07:50.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T01:07:50.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T01:07:50.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T01:07:50.340 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T01:07:50.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T01:07:50.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T01:07:50.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T01:07:50.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T01:07:50.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T01:07:50.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T01:07:50.341 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T01:07:50.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T01:07:50.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T01:07:50.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T01:07:50.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T01:07:50.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-06-23T01:07:50.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-06-23T01:07:50.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-06-23T01:07:50.342 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-06-23T01:07:50.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-06-23T01:07:50.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-06-23T01:07:50.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-06-23T01:07:50.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-06-23T01:07:50.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-06-23T01:07:50.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-06-23T01:07:50.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-06-23T01:07:50.343 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-06-23T01:07:50.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-06-23T01:07:50.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-06-23T01:07:50.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-06-23T01:07:50.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-06-23T01:07:50.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-06-23T01:07:50.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-06-23T01:07:50.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-06-23T01:07:50.344 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-06-23T01:07:50.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-06-23T01:07:50.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-06-23T01:07:50.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-06-23T01:07:50.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-06-23T01:07:50.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-06-23T01:07:50.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-06-23T01:07:50.345 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-06-23T01:07:50.346 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-06-23T01:07:50.346 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-06-23T01:07:50.346 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-06-23T01:07:50.346 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-06-23T01:07:50.346 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-06-23T01:07:50.346 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-06-23T01:07:50.346 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-06-23T01:07:50.346 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-06-23T01:07:50.347 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-06-23T01:07:50.347 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-06-23T01:07:50.347 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-06-23T01:07:50.347 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-06-23T01:07:50.347 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-06-23T01:07:50.347 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-06-23T01:07:50.347 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-06-23T01:07:50.347 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-06-23T01:07:50.348 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-06-23T01:07:50.348 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-06-23T01:07:50.348 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-06-23T01:07:50.348 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-06-23T01:07:50.348 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T01:07:50.348 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T01:07:50.348 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T01:07:50.349 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T01:07:50.504 INFO:teuthology.orchestra.run.smithi039.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T01:07:50.514 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T01:07:50.514 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:07:50.682 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:07:50.682 INFO:teuthology.orchestra.run.smithi039.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:07:51.177 INFO:teuthology.orchestra.run.smithi039.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:07:51.178 INFO:teuthology.orchestra.run.smithi039.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T01:07:51.178 INFO:teuthology.orchestra.run.smithi039.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T01:07:51.178 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:51.514 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T01:07:51.514 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T01:07:51.514 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T01:07:51.514 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:07:51.515 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T01:07:51.515 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:07:51.515 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:07:51.515 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:07:51.515 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T01:07:51.515 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:07:51.515 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T01:07:51.516 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T01:07:51.516 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:07:51.516 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T01:07:51.516 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T01:07:51.516 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T01:07:51.517 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T01:07:51.517 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T01:07:51.517 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T01:07:51.517 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T01:07:51.517 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T01:07:51.517 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T01:07:51.525 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T01:07:51.525 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T01:07:51.525 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T01:07:51.525 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T01:07:51.525 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T01:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T01:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T01:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T01:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T01:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T01:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T01:07:51.526 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T01:07:51.527 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T01:07:51.527 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T01:07:51.527 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T01:07:51.527 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T01:07:51.527 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T01:07:51.527 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T01:07:51.527 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T01:07:51.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T01:07:51.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T01:07:51.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T01:07:51.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T01:07:51.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T01:07:51.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T01:07:51.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T01:07:51.528 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T01:07:51.529 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T01:07:51.529 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T01:07:51.529 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T01:07:51.529 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T01:07:51.529 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T01:07:51.529 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T01:07:51.529 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T01:07:51.530 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T01:07:51.531 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T01:07:51.531 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T01:07:51.531 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T01:07:51.531 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T01:07:51.531 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T01:07:51.531 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T01:07:51.532 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-06-23T01:07:51.532 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-06-23T01:07:51.532 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-06-23T01:07:51.532 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-06-23T01:07:51.532 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-06-23T01:07:51.532 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-06-23T01:07:51.532 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-06-23T01:07:51.532 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-06-23T01:07:51.533 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-06-23T01:07:51.533 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-06-23T01:07:51.533 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-06-23T01:07:51.533 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-06-23T01:07:51.533 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-06-23T01:07:51.533 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-06-23T01:07:51.533 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-06-23T01:07:51.533 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-06-23T01:07:51.534 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-06-23T01:07:51.534 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-06-23T01:07:51.534 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-06-23T01:07:51.534 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-06-23T01:07:51.534 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-06-23T01:07:51.534 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-06-23T01:07:51.534 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-06-23T01:07:51.535 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-06-23T01:07:51.535 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-06-23T01:07:51.535 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-06-23T01:07:51.535 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-06-23T01:07:51.535 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-06-23T01:07:51.535 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-06-23T01:07:51.535 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-06-23T01:07:51.535 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-06-23T01:07:51.536 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-06-23T01:07:51.536 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-06-23T01:07:51.536 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-06-23T01:07:51.536 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-06-23T01:07:51.536 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-06-23T01:07:51.536 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-06-23T01:07:51.536 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-06-23T01:07:51.536 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-06-23T01:07:51.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-06-23T01:07:51.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-06-23T01:07:51.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-06-23T01:07:51.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-06-23T01:07:51.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-06-23T01:07:51.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-06-23T01:07:51.537 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-06-23T01:07:51.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-06-23T01:07:51.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T01:07:51.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T01:07:51.538 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T01:07:52.480 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:07:52.480 INFO:teuthology.orchestra.run.smithi073.stdout:Last metadata expiration check: 0:01:45 ago on Fri 23 Jun 2023 01:06:07 AM UTC. 2023-06-23T01:07:53.817 INFO:teuthology.orchestra.run.smithi039.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:07:53.817 INFO:teuthology.orchestra.run.smithi039.stdout:Last metadata expiration check: 0:01:46 ago on Fri 23 Jun 2023 01:06:07 AM UTC. 2023-06-23T01:07:54.697 INFO:teuthology.orchestra.run.smithi073.stdout:Installed products updated. 2023-06-23T01:07:54.824 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:07:54.825 INFO:teuthology.orchestra.run.smithi073.stdout:Upgraded: 2023-06-23T01:07:54.825 INFO:teuthology.orchestra.run.smithi073.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.825 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.825 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2023-06-23T01:07:54.825 INFO:teuthology.orchestra.run.smithi073.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T01:07:54.825 INFO:teuthology.orchestra.run.smithi073.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T01:07:54.825 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.826 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.826 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.826 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.826 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.826 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.826 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.826 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.826 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.827 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.827 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.827 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.827 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.827 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.827 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.827 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.827 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.828 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.828 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.828 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.828 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.828 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.828 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:54.828 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T01:07:54.829 INFO:teuthology.orchestra.run.smithi073.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T01:07:54.829 INFO:teuthology.orchestra.run.smithi073.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T01:07:54.829 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T01:07:54.829 INFO:teuthology.orchestra.run.smithi073.stdout: jq-1.5-12.el8.x86_64 2023-06-23T01:07:54.829 INFO:teuthology.orchestra.run.smithi073.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T01:07:54.829 INFO:teuthology.orchestra.run.smithi073.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T01:07:54.829 INFO:teuthology.orchestra.run.smithi073.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T01:07:54.829 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.830 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.830 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.830 INFO:teuthology.orchestra.run.smithi073.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T01:07:54.830 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T01:07:54.830 INFO:teuthology.orchestra.run.smithi073.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T01:07:54.830 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T01:07:54.830 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T01:07:54.831 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.831 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.831 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T01:07:54.831 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.831 INFO:teuthology.orchestra.run.smithi073.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T01:07:54.831 INFO:teuthology.orchestra.run.smithi073.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T01:07:54.831 INFO:teuthology.orchestra.run.smithi073.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T01:07:54.832 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T01:07:54.832 INFO:teuthology.orchestra.run.smithi073.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T01:07:54.832 INFO:teuthology.orchestra.run.smithi073.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T01:07:54.832 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T01:07:54.832 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T01:07:54.832 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T01:07:54.832 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T01:07:54.832 INFO:teuthology.orchestra.run.smithi073.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T01:07:54.833 INFO:teuthology.orchestra.run.smithi073.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T01:07:54.833 INFO:teuthology.orchestra.run.smithi073.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T01:07:54.833 INFO:teuthology.orchestra.run.smithi073.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T01:07:54.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T01:07:54.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T01:07:54.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T01:07:54.833 INFO:teuthology.orchestra.run.smithi073.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T01:07:54.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T01:07:54.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T01:07:54.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T01:07:54.834 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T01:07:54.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T01:07:54.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T01:07:54.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T01:07:54.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T01:07:54.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T01:07:54.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T01:07:54.835 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T01:07:54.836 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T01:07:54.836 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T01:07:54.836 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T01:07:54.836 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T01:07:54.836 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T01:07:54.836 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T01:07:54.836 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T01:07:54.837 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T01:07:54.837 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T01:07:54.837 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T01:07:54.837 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T01:07:54.837 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T01:07:54.837 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T01:07:54.837 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T01:07:54.837 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T01:07:54.838 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.838 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.838 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T01:07:54.838 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T01:07:54.838 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.838 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T01:07:54.838 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T01:07:54.839 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T01:07:54.839 INFO:teuthology.orchestra.run.smithi073.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T01:07:54.839 INFO:teuthology.orchestra.run.smithi073.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T01:07:54.839 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T01:07:54.839 INFO:teuthology.orchestra.run.smithi073.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T01:07:54.839 INFO:teuthology.orchestra.run.smithi073.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T01:07:54.839 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T01:07:54.840 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T01:07:54.840 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T01:07:54.840 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T01:07:54.840 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T01:07:54.840 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.840 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.840 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:54.841 INFO:teuthology.orchestra.run.smithi073.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T01:07:54.841 INFO:teuthology.orchestra.run.smithi073.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T01:07:54.841 INFO:teuthology.orchestra.run.smithi073.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T01:07:54.841 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T01:07:54.841 INFO:teuthology.orchestra.run.smithi073.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T01:07:54.842 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:07:54.842 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2023-06-23T01:07:55.035 DEBUG:teuthology.parallel:result is None 2023-06-23T01:07:56.056 INFO:teuthology.orchestra.run.smithi039.stdout:Installed products updated. 2023-06-23T01:07:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:56.190 INFO:teuthology.orchestra.run.smithi039.stdout:Upgraded: 2023-06-23T01:07:56.190 INFO:teuthology.orchestra.run.smithi039.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.191 INFO:teuthology.orchestra.run.smithi039.stdout:Installed: 2023-06-23T01:07:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T01:07:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T01:07:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.191 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.192 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.193 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:07:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T01:07:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T01:07:56.194 INFO:teuthology.orchestra.run.smithi039.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T01:07:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T01:07:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: jq-1.5-12.el8.x86_64 2023-06-23T01:07:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T01:07:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T01:07:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T01:07:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.195 INFO:teuthology.orchestra.run.smithi039.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T01:07:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T01:07:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T01:07:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T01:07:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T01:07:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.196 INFO:teuthology.orchestra.run.smithi039.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T01:07:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T01:07:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T01:07:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T01:07:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T01:07:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T01:07:56.197 INFO:teuthology.orchestra.run.smithi039.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T01:07:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T01:07:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T01:07:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T01:07:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T01:07:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T01:07:56.198 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T01:07:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T01:07:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T01:07:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T01:07:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T01:07:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T01:07:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T01:07:56.199 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T01:07:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T01:07:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T01:07:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T01:07:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T01:07:56.200 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T01:07:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T01:07:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T01:07:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T01:07:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T01:07:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T01:07:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T01:07:56.201 INFO:teuthology.orchestra.run.smithi039.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T01:07:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T01:07:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T01:07:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T01:07:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T01:07:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T01:07:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T01:07:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T01:07:56.202 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T01:07:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T01:07:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T01:07:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T01:07:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T01:07:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T01:07:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.203 INFO:teuthology.orchestra.run.smithi039.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T01:07:56.204 INFO:teuthology.orchestra.run.smithi039.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T01:07:56.204 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.204 INFO:teuthology.orchestra.run.smithi039.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T01:07:56.204 INFO:teuthology.orchestra.run.smithi039.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T01:07:56.204 INFO:teuthology.orchestra.run.smithi039.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T01:07:56.205 INFO:teuthology.orchestra.run.smithi039.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T01:07:56.205 INFO:teuthology.orchestra.run.smithi039.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T01:07:56.205 INFO:teuthology.orchestra.run.smithi039.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T01:07:56.206 INFO:teuthology.orchestra.run.smithi039.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T01:07:56.206 INFO:teuthology.orchestra.run.smithi039.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T01:07:56.206 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T01:07:56.206 INFO:teuthology.orchestra.run.smithi039.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T01:07:56.206 INFO:teuthology.orchestra.run.smithi039.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T01:07:56.206 INFO:teuthology.orchestra.run.smithi039.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T01:07:56.206 INFO:teuthology.orchestra.run.smithi039.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T01:07:56.207 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.207 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.207 INFO:teuthology.orchestra.run.smithi039.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:07:56.207 INFO:teuthology.orchestra.run.smithi039.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T01:07:56.207 INFO:teuthology.orchestra.run.smithi039.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T01:07:56.207 INFO:teuthology.orchestra.run.smithi039.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T01:07:56.207 INFO:teuthology.orchestra.run.smithi039.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T01:07:56.208 INFO:teuthology.orchestra.run.smithi039.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T01:07:56.208 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:56.208 INFO:teuthology.orchestra.run.smithi039.stdout:Complete! 2023-06-23T01:07:56.410 DEBUG:teuthology.parallel:result is None 2023-06-23T01:07:56.410 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:07:56.554 DEBUG:teuthology.orchestra.run.smithi039:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T01:07:56.580 INFO:teuthology.orchestra.run.smithi039.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T01:07:56.581 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T01:07:56.582 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T01:07:56.582 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:07:56.718 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T01:07:56.741 INFO:teuthology.orchestra.run.smithi073.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T01:07:56.741 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T01:07:56.741 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T01:07:56.742 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-23T01:07:56.743 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:07:56.743 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T01:07:56.783 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:07:56.784 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T01:07:56.824 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-23T01:07:56.824 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:07:56.825 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T01:07:56.850 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T01:07:56.916 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:07:56.917 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T01:07:56.942 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T01:07:57.006 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-23T01:07:57.007 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:07:57.007 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T01:07:57.033 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T01:07:57.099 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:07:57.100 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T01:07:57.125 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T01:07:57.189 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-23T01:07:57.190 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:07:57.190 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T01:07:57.217 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T01:07:57.285 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:07:57.285 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T01:07:57.309 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T01:07:57.375 INFO:teuthology.run_tasks:Running task ceph... 2023-06-23T01:07:57.442 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-06-23T01:07:57.443 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 777 /var/log/ceph 2023-06-23T01:07:57.445 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /var/log/ceph 2023-06-23T01:07:57.470 INFO:tasks.ceph:Disabling ceph logrotate... 2023-06-23T01:07:57.470 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T01:07:57.510 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T01:07:57.535 INFO:tasks.ceph:Creating extra log directories... 2023-06-23T01:07:57.536 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T01:07:57.577 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T01:07:57.607 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-06-23T01:07:57.608 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-06-23T01:07:57.609 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-06-23T01:07:57.609 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:07:57.609 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T01:07:57.656 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T01:07:57.774 DEBUG:teuthology.orchestra.run.smithi039:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T01:07:57.800 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:07:57.800 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T01:07:57.814 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T01:07:57.929 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T01:07:57.969 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-06-23T01:07:57.970 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'inconsistent rstat on inode', 'Error recovering journal'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-06-23T01:07:57.970 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920', 'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7311920', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'bad backtrace', 'object missing on disk', 'error reading table object', 'error reading sessionmap', 'unmatched fragstat', 'unmatched rstat', 'was unreadable, recreating it now', 'Scrub error on inode', 'Metadata damage detected', 'inconsistent rstat on inode', 'Error recovering journal'], 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'kclient': {'syntax': 'v2'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9037, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'targets': {'smithi039.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA+FEBb0lM2JOjErPpPnTaufA95sbkevpUNdZRQxeWME9cdgBhMzKCyPEvUN+ik/0QBSqca9d17MEbfEIIxoqNw=', 'smithi073.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiGlkx9VorGvQLNcRASuoLUUj4Sv96vdo3OTdVFt9DQkbdF27RlK8Tj2JUas2bUAA8yZ89j7Io0YejSM2nYka4iKTaGqgBVxa4tHUZ+d4PT8GrVn3U/U91xQCGU2+8rgcNAWMPYAC+e4xor6VjopW1+k6DKiC2tc0pL+tMD5EpPA1VVUciLB6I3gSD0Y9Dcjf2mNIzaxnDl+CW6oUcyfq/TGw6tsluKyr14CE6ZELMn1kvUM/zH/P5uQKUiQyhYKzy5JnUzlawpMUQvpCvN1AlqlY1lVyPEogq6bCRl7Rkn+61H4uA85igg/9l29bMuHi4uVHnB+Q02Em6+aVr4/PUnOOyOdvQopTeZm/ZBHYnP5gSo3I+Kk6gr+VGT+lG7F8lBaqzvCCBOWQzLXyuEAuZRwhM5+S2MNEbU5xXPMr3XGp3GrPvCZbzZDmJMqypLlRB7f26kRXilC26fl5ODcmPEK3ecNT+JMwjUxGJCmWiz/BYXrPBZCR4LdJl0OqDqEM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_data_scan']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '076bbebc42a14f7d568aaa78eabb0038327bcb23', 'timestamp': '2023-06-22_15:25:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-06-23T01:07:57.971 DEBUG:teuthology.orchestra.run.smithi039:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T01:07:57.974 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T01:07:58.018 DEBUG:teuthology.orchestra.run.smithi039:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T01:07:58.030 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T01:07:58.083 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:07:58.083 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:07:58.110 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:07:58.111 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_1 2023-06-23T01:07:58.165 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:07:58.165 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:07:58.165 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 212192 Links: 1 2023-06-23T01:07:58.166 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:07:58.166 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:07:58.166 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-06-23 01:07:51.087170210 +0000 2023-06-23T01:07:58.166 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-06-23 01:05:20.458818147 +0000 2023-06-23T01:07:58.166 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-06-23 01:05:20.458818147 +0000 2023-06-23T01:07:58.166 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-06-23T01:07:58.167 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:07:58.230 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-06-23T01:07:58.231 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-06-23T01:07:58.231 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000291392 s, 1.8 MB/s 2023-06-23T01:07:58.232 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:07:58.288 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_2 2023-06-23T01:07:58.344 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:07:58.345 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:07:58.345 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 215254 Links: 1 2023-06-23T01:07:58.345 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:07:58.345 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:07:58.345 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-06-23 01:07:51.088170190 +0000 2023-06-23T01:07:58.345 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-06-23 01:05:20.968807604 +0000 2023-06-23T01:07:58.345 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-06-23 01:05:20.968807604 +0000 2023-06-23T01:07:58.346 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-06-23T01:07:58.346 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:07:58.409 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-06-23T01:07:58.410 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-06-23T01:07:58.410 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000290791 s, 1.8 MB/s 2023-06-23T01:07:58.411 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:07:58.466 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_3 2023-06-23T01:07:58.522 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:07:58.522 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:07:58.522 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 215357 Links: 1 2023-06-23T01:07:58.522 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:07:58.522 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:07:58.523 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-06-23 01:07:51.088170190 +0000 2023-06-23T01:07:58.523 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-06-23 01:05:21.493796750 +0000 2023-06-23T01:07:58.523 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-06-23 01:05:21.493796750 +0000 2023-06-23T01:07:58.523 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-06-23T01:07:58.524 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:07:58.588 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-06-23T01:07:58.589 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-06-23T01:07:58.589 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000342207 s, 1.5 MB/s 2023-06-23T01:07:58.590 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:07:58.647 DEBUG:teuthology.orchestra.run.smithi039:> stat /dev/vg_nvme/lv_4 2023-06-23T01:07:58.702 INFO:teuthology.orchestra.run.smithi039.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:07:58.703 INFO:teuthology.orchestra.run.smithi039.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:07:58.703 INFO:teuthology.orchestra.run.smithi039.stdout:Device: 6h/6d Inode: 215506 Links: 1 2023-06-23T01:07:58.703 INFO:teuthology.orchestra.run.smithi039.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:07:58.703 INFO:teuthology.orchestra.run.smithi039.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:07:58.703 INFO:teuthology.orchestra.run.smithi039.stdout:Access: 2023-06-23 01:07:51.088170190 +0000 2023-06-23T01:07:58.703 INFO:teuthology.orchestra.run.smithi039.stdout:Modify: 2023-06-23 01:05:22.019785877 +0000 2023-06-23T01:07:58.703 INFO:teuthology.orchestra.run.smithi039.stdout:Change: 2023-06-23 01:05:22.019785877 +0000 2023-06-23T01:07:58.704 INFO:teuthology.orchestra.run.smithi039.stdout: Birth: - 2023-06-23T01:07:58.704 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:07:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records in 2023-06-23T01:07:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:1+0 records out 2023-06-23T01:07:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:512 bytes copied, 0.000280957 s, 1.8 MB/s 2023-06-23T01:07:58.769 DEBUG:teuthology.orchestra.run.smithi039:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:07:58.827 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:07:58.828 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:07:58.828 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:07:58.843 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:07:58.844 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_1 2023-06-23T01:07:58.899 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:07:58.899 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:07:58.899 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 6h/6d Inode: 204690 Links: 1 2023-06-23T01:07:58.900 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:07:58.900 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:07:58.900 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2023-06-23 01:07:49.899617450 +0000 2023-06-23T01:07:58.900 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2023-06-23 00:58:37.813361856 +0000 2023-06-23T01:07:58.900 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2023-06-23 00:58:37.813361856 +0000 2023-06-23T01:07:58.900 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: - 2023-06-23T01:07:58.901 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:07:58.963 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2023-06-23T01:07:58.963 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2023-06-23T01:07:58.963 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000245924 s, 2.1 MB/s 2023-06-23T01:07:58.964 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:07:59.019 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_2 2023-06-23T01:07:59.074 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:07:59.074 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:07:59.074 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 6h/6d Inode: 213340 Links: 1 2023-06-23T01:07:59.074 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:07:59.074 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:07:59.074 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2023-06-23 01:07:49.899617450 +0000 2023-06-23T01:07:59.075 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2023-06-23 00:58:38.322349073 +0000 2023-06-23T01:07:59.075 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2023-06-23 00:58:38.322349073 +0000 2023-06-23T01:07:59.075 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: - 2023-06-23T01:07:59.075 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:07:59.140 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2023-06-23T01:07:59.141 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2023-06-23T01:07:59.141 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000226166 s, 2.3 MB/s 2023-06-23T01:07:59.141 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:07:59.196 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_3 2023-06-23T01:07:59.251 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:07:59.251 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:07:59.251 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 6h/6d Inode: 207845 Links: 1 2023-06-23T01:07:59.251 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:07:59.251 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:07:59.252 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2023-06-23 01:07:49.899617450 +0000 2023-06-23T01:07:59.252 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2023-06-23 00:58:38.847335888 +0000 2023-06-23T01:07:59.252 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2023-06-23 00:58:38.847335888 +0000 2023-06-23T01:07:59.252 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: - 2023-06-23T01:07:59.252 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:07:59.315 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2023-06-23T01:07:59.315 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2023-06-23T01:07:59.315 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.00020877 s, 2.5 MB/s 2023-06-23T01:07:59.316 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:07:59.371 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_4 2023-06-23T01:07:59.426 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:07:59.426 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:07:59.427 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 6h/6d Inode: 213582 Links: 1 2023-06-23T01:07:59.427 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:07:59.427 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:07:59.427 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2023-06-23 01:07:49.899617450 +0000 2023-06-23T01:07:59.427 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2023-06-23 00:58:39.356323105 +0000 2023-06-23T01:07:59.427 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2023-06-23 00:58:39.356323105 +0000 2023-06-23T01:07:59.427 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: - 2023-06-23T01:07:59.428 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:07:59.489 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2023-06-23T01:07:59.489 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2023-06-23T01:07:59.489 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000213169 s, 2.4 MB/s 2023-06-23T01:07:59.490 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:07:59.547 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:07:59.548 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T01:07:59.549 INFO:tasks.ceph:Generating config... 2023-06-23T01:07:59.552 INFO:tasks.ceph:[client] client mount timeout = 600 2023-06-23T01:07:59.552 INFO:tasks.ceph:[client] debug client = 20 2023-06-23T01:07:59.552 INFO:tasks.ceph:[client] debug ms = 1 2023-06-23T01:07:59.553 INFO:tasks.ceph:[client] pid file = 2023-06-23T01:07:59.553 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-06-23T01:07:59.553 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-06-23T01:07:59.554 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-06-23T01:07:59.554 INFO:tasks.ceph:[mds] debug mds = 20 2023-06-23T01:07:59.555 INFO:tasks.ceph:[mds] debug ms = 1 2023-06-23T01:07:59.555 INFO:tasks.ceph:[mds] mds debug frag = True 2023-06-23T01:07:59.556 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-06-23T01:07:59.557 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-06-23T01:07:59.557 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-06-23T01:07:59.557 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-06-23T01:07:59.558 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-06-23T01:07:59.558 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-06-23T01:07:59.558 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-06-23T01:07:59.558 INFO:tasks.ceph:[mgr] debug ms = 1 2023-06-23T01:07:59.559 INFO:tasks.ceph:[mon] debug mon = 20 2023-06-23T01:07:59.559 INFO:tasks.ceph:[mon] debug ms = 1 2023-06-23T01:07:59.559 INFO:tasks.ceph:[mon] debug paxos = 20 2023-06-23T01:07:59.559 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-06-23T01:07:59.559 INFO:tasks.ceph:[osd] bdev async discard = True 2023-06-23T01:07:59.560 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-06-23T01:07:59.560 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-06-23T01:07:59.560 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-06-23T01:07:59.561 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-06-23T01:07:59.561 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-06-23T01:07:59.561 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-06-23T01:07:59.561 INFO:tasks.ceph:[osd] debug ms = 1 2023-06-23T01:07:59.562 INFO:tasks.ceph:[osd] debug osd = 20 2023-06-23T01:07:59.562 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-06-23T01:07:59.562 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-06-23T01:07:59.562 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-06-23T01:07:59.562 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-06-23T01:07:59.563 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-06-23T01:07:59.563 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-06-23T01:07:59.563 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-06-23T01:07:59.563 INFO:tasks.ceph:Setting up mon.a... 2023-06-23T01:07:59.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-06-23T01:07:59.621 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.keyring 2023-06-23T01:07:59.624 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-06-23T01:07:59.668 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:07:59.734 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.39'), ('mon.b', '172.21.15.73'), ('mon.c', '[v2:172.21.15.73:3301,v1:172.21.15.73:6790]')] 2023-06-23T01:07:59.734 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.39,172.21.15.73,[v2:172.21.15.73:3301,v1:172.21.15.73:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-06-23T01:07:59.735 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:07:59.736 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T01:07:59.788 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.39 --add b 172.21.15.73 --addv c '[v2:172.21.15.73:3301,v1:172.21.15.73:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:07:59.874 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:07:59.874 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: generated fsid 314a4b16-8059-4f2c-986c-1624c6de1afb 2023-06-23T01:07:59.874 INFO:teuthology.orchestra.run.smithi039.stdout:setting min_mon_release = octopus 2023-06-23T01:07:59.874 INFO:teuthology.orchestra.run.smithi039.stdout:epoch 0 2023-06-23T01:07:59.875 INFO:teuthology.orchestra.run.smithi039.stdout:fsid 314a4b16-8059-4f2c-986c-1624c6de1afb 2023-06-23T01:07:59.875 INFO:teuthology.orchestra.run.smithi039.stdout:last_changed 2023-06-23T01:07:59.871934+0000 2023-06-23T01:07:59.875 INFO:teuthology.orchestra.run.smithi039.stdout:created 2023-06-23T01:07:59.871934+0000 2023-06-23T01:07:59.875 INFO:teuthology.orchestra.run.smithi039.stdout:min_mon_release 15 (octopus) 2023-06-23T01:07:59.875 INFO:teuthology.orchestra.run.smithi039.stdout:election_strategy: 1 2023-06-23T01:07:59.875 INFO:teuthology.orchestra.run.smithi039.stdout:0: [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] mon.a 2023-06-23T01:07:59.875 INFO:teuthology.orchestra.run.smithi039.stdout:1: [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon.b 2023-06-23T01:07:59.875 INFO:teuthology.orchestra.run.smithi039.stdout:2: [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] mon.c 2023-06-23T01:07:59.876 INFO:teuthology.orchestra.run.smithi039.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-06-23T01:07:59.877 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T01:07:59.893 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 314a4b16-8059-4f2c-986c-1624c6de1afb... 2023-06-23T01:07:59.894 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T01:07:59.935 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T01:07:59.979 INFO:teuthology.orchestra.run.smithi039.stdout:[global] 2023-06-23T01:07:59.979 INFO:teuthology.orchestra.run.smithi039.stdout: chdir = "" 2023-06-23T01:07:59.979 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T01:07:59.979 INFO:teuthology.orchestra.run.smithi039.stdout: auth supported = cephx 2023-06-23T01:07:59.979 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.980 INFO:teuthology.orchestra.run.smithi039.stdout: filestore xattr use omap = true 2023-06-23T01:07:59.980 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.980 INFO:teuthology.orchestra.run.smithi039.stdout: mon clock drift allowed = 1.000 2023-06-23T01:07:59.980 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.980 INFO:teuthology.orchestra.run.smithi039.stdout: osd crush chooseleaf type = 0 2023-06-23T01:07:59.980 INFO:teuthology.orchestra.run.smithi039.stdout: auth debug = true 2023-06-23T01:07:59.980 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.981 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on old message = true 2023-06-23T01:07:59.981 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on bug = true 2023-06-23T01:07:59.981 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.981 INFO:teuthology.orchestra.run.smithi039.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T01:07:59.981 INFO:teuthology.orchestra.run.smithi039.stdout: mon pg warn max object skew = 0 2023-06-23T01:07:59.981 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.981 INFO:teuthology.orchestra.run.smithi039.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T01:07:59.981 INFO:teuthology.orchestra.run.smithi039.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T01:07:59.982 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.982 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default size = 2 2023-06-23T01:07:59.982 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.982 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow primary affinity = true 2023-06-23T01:07:59.982 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd allow pg remap = true 2023-06-23T01:07:59.982 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on legacy crush tunables = false 2023-06-23T01:07:59.982 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on crush straw calc version zero = false 2023-06-23T01:07:59.983 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on no sortbitwise = false 2023-06-23T01:07:59.983 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on osd down out interval zero = false 2023-06-23T01:07:59.983 INFO:teuthology.orchestra.run.smithi039.stdout: mon warn on too few osds = false 2023-06-23T01:07:59.983 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T01:07:59.983 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T01:07:59.983 INFO:teuthology.orchestra.run.smithi039.stdout: mon_allow_pool_size_one = true 2023-06-23T01:07:59.983 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.983 INFO:teuthology.orchestra.run.smithi039.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T01:07:59.984 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.984 INFO:teuthology.orchestra.run.smithi039.stdout: osd default data pool replay window = 5 2023-06-23T01:07:59.985 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.985 INFO:teuthology.orchestra.run.smithi039.stdout: mon allow pool delete = true 2023-06-23T01:07:59.986 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.986 INFO:teuthology.orchestra.run.smithi039.stdout: mon cluster log file level = debug 2023-06-23T01:07:59.986 INFO:teuthology.orchestra.run.smithi039.stdout: debug asserts on shutdown = true 2023-06-23T01:07:59.986 INFO:teuthology.orchestra.run.smithi039.stdout: mon health detail to clog = false 2023-06-23T01:07:59.986 INFO:teuthology.orchestra.run.smithi039.stdout: mon host = "172.21.15.39,172.21.15.73,[v2:172.21.15.73:3301,v1:172.21.15.73:6790]" 2023-06-23T01:07:59.986 INFO:teuthology.orchestra.run.smithi039.stdout: ms die on skipped message = False 2023-06-23T01:07:59.987 INFO:teuthology.orchestra.run.smithi039.stdout: fsid = 314a4b16-8059-4f2c-986c-1624c6de1afb 2023-06-23T01:07:59.987 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.987 INFO:teuthology.orchestra.run.smithi039.stdout:[osd] 2023-06-23T01:07:59.987 INFO:teuthology.orchestra.run.smithi039.stdout: osd journal size = 100 2023-06-23T01:07:59.987 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.987 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub load threshold = 5.0 2023-06-23T01:07:59.987 INFO:teuthology.orchestra.run.smithi039.stdout: osd scrub max interval = 600 2023-06-23T01:07:59.988 INFO:teuthology.orchestra.run.smithi039.stdout: osd mclock profile = high_recovery_ops 2023-06-23T01:07:59.988 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.988 INFO:teuthology.orchestra.run.smithi039.stdout: osd recover clone overlap = true 2023-06-23T01:07:59.988 INFO:teuthology.orchestra.run.smithi039.stdout: osd recovery max chunk = 1048576 2023-06-23T01:07:59.988 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.988 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug shutdown = true 2023-06-23T01:07:59.988 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug op order = true 2023-06-23T01:07:59.988 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug verify stray on activate = true 2023-06-23T01:07:59.989 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.989 INFO:teuthology.orchestra.run.smithi039.stdout: osd open classes on start = true 2023-06-23T01:07:59.989 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug pg log writeout = true 2023-06-23T01:07:59.989 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.989 INFO:teuthology.orchestra.run.smithi039.stdout: osd deep scrub update digest min age = 30 2023-06-23T01:07:59.989 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.989 INFO:teuthology.orchestra.run.smithi039.stdout: osd map max advance = 10 2023-06-23T01:07:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: journal zero on create = true 2023-06-23T01:07:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: filestore ondisk finisher threads = 3 2023-06-23T01:07:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: filestore apply finisher threads = 3 2023-06-23T01:07:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: bdev debug aio = true 2023-06-23T01:07:59.990 INFO:teuthology.orchestra.run.smithi039.stdout: osd debug misdirected ops = true 2023-06-23T01:07:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: bdev async discard = True 2023-06-23T01:07:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: bdev enable discard = True 2023-06-23T01:07:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore allocator = bitmap 2023-06-23T01:07:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore block size = 96636764160 2023-06-23T01:07:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: bluestore fsck on mount = True 2023-06-23T01:07:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluefs = 1/20 2023-06-23T01:07:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: debug bluestore = 1/20 2023-06-23T01:07:59.991 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-06-23T01:07:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: debug osd = 20 2023-06-23T01:07:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: debug rocksdb = 4/10 2023-06-23T01:07:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T01:07:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd full ratio = 0.9 2023-06-23T01:07:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd nearfull ratio = 0.8 2023-06-23T01:07:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: osd failsafe full ratio = 0.95 2023-06-23T01:07:59.992 INFO:teuthology.orchestra.run.smithi039.stdout: osd objectstore = bluestore 2023-06-23T01:07:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2023-06-23T01:07:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.993 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr] 2023-06-23T01:07:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-06-23T01:07:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: debug mgr = 20 2023-06-23T01:07:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2023-06-23T01:07:59.993 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2023-06-23T01:07:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:07:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:07:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: mgr/telemetry/nag = false 2023-06-23T01:07:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.994 INFO:teuthology.orchestra.run.smithi039.stdout:[mon] 2023-06-23T01:07:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-06-23T01:07:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: debug mon = 20 2023-06-23T01:07:59.994 INFO:teuthology.orchestra.run.smithi039.stdout: debug paxos = 20 2023-06-23T01:07:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: debug auth = 20 2023-06-23T01:07:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: mon data avail warn = 5 2023-06-23T01:07:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: mon mgr mkfs grace = 240 2023-06-23T01:07:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:07:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd reporter subtree level = osd 2023-06-23T01:07:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: mon osd prime pg temp = true 2023-06-23T01:07:59.995 INFO:teuthology.orchestra.run.smithi039.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:07:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T01:07:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T01:07:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T01:07:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T01:07:59.996 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T01:07:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T01:07:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: # 1m isn't quite enough 2023-06-23T01:07:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: mon_down_mkfs_grace = 2m 2023-06-23T01:07:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: mon_warn_on_filestore_osds = false 2023-06-23T01:07:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: mon op complaint time = 120 2023-06-23T01:07:59.997 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:07:59.998 INFO:teuthology.orchestra.run.smithi039.stdout:[client] 2023-06-23T01:07:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: rgw cache enabled = true 2023-06-23T01:07:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable ops log = true 2023-06-23T01:07:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: rgw enable usage log = true 2023-06-23T01:07:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T01:07:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T01:07:59.998 INFO:teuthology.orchestra.run.smithi039.stdout: client mount timeout = 600 2023-06-23T01:07:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: debug client = 20 2023-06-23T01:07:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-06-23T01:07:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: pid file = "" 2023-06-23T01:07:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2023-06-23T01:07:59.999 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2023-06-23T01:07:59.999 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.a] 2023-06-23T01:07:59.999 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.b] 2023-06-23T01:07:59.999 INFO:teuthology.orchestra.run.smithi039.stdout:[mon.c] 2023-06-23T01:08:00.000 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2023-06-23T01:08:00.000 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2023-06-23T01:08:00.000 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2023-06-23T01:08:00.000 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2023-06-23T01:08:00.000 INFO:teuthology.orchestra.run.smithi039.stdout:[mds] 2023-06-23T01:08:00.000 INFO:teuthology.orchestra.run.smithi039.stdout: debug mds = 20 2023-06-23T01:08:00.000 INFO:teuthology.orchestra.run.smithi039.stdout: debug ms = 1 2023-06-23T01:08:00.000 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug frag = True 2023-06-23T01:08:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: mds debug scatterstat = True 2023-06-23T01:08:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: mds op complaint time = 180 2023-06-23T01:08:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: mds verify scatter = True 2023-06-23T01:08:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: osd op complaint time = 180 2023-06-23T01:08:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: rados mon op timeout = 900 2023-06-23T01:08:00.001 INFO:teuthology.orchestra.run.smithi039.stdout: rados osd op timeout = 900 2023-06-23T01:08:00.003 INFO:teuthology.orchestra.run.smithi073.stdout:[global] 2023-06-23T01:08:00.003 INFO:teuthology.orchestra.run.smithi073.stdout: chdir = "" 2023-06-23T01:08:00.003 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T01:08:00.003 INFO:teuthology.orchestra.run.smithi073.stdout: auth supported = cephx 2023-06-23T01:08:00.003 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.004 INFO:teuthology.orchestra.run.smithi073.stdout: filestore xattr use omap = true 2023-06-23T01:08:00.004 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.004 INFO:teuthology.orchestra.run.smithi073.stdout: mon clock drift allowed = 1.000 2023-06-23T01:08:00.004 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.004 INFO:teuthology.orchestra.run.smithi073.stdout: osd crush chooseleaf type = 0 2023-06-23T01:08:00.004 INFO:teuthology.orchestra.run.smithi073.stdout: auth debug = true 2023-06-23T01:08:00.004 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.005 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on old message = true 2023-06-23T01:08:00.005 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on bug = true 2023-06-23T01:08:00.005 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.005 INFO:teuthology.orchestra.run.smithi073.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T01:08:00.005 INFO:teuthology.orchestra.run.smithi073.stdout: mon pg warn max object skew = 0 2023-06-23T01:08:00.005 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.005 INFO:teuthology.orchestra.run.smithi073.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T01:08:00.005 INFO:teuthology.orchestra.run.smithi073.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T01:08:00.006 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.006 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default size = 2 2023-06-23T01:08:00.006 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.006 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow primary affinity = true 2023-06-23T01:08:00.006 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow pg remap = true 2023-06-23T01:08:00.006 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on legacy crush tunables = false 2023-06-23T01:08:00.006 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on crush straw calc version zero = false 2023-06-23T01:08:00.006 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on no sortbitwise = false 2023-06-23T01:08:00.007 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on osd down out interval zero = false 2023-06-23T01:08:00.007 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on too few osds = false 2023-06-23T01:08:00.007 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T01:08:00.007 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T01:08:00.007 INFO:teuthology.orchestra.run.smithi073.stdout: mon_allow_pool_size_one = true 2023-06-23T01:08:00.007 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.007 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T01:08:00.007 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.008 INFO:teuthology.orchestra.run.smithi073.stdout: osd default data pool replay window = 5 2023-06-23T01:08:00.008 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.008 INFO:teuthology.orchestra.run.smithi073.stdout: mon allow pool delete = true 2023-06-23T01:08:00.008 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.008 INFO:teuthology.orchestra.run.smithi073.stdout: mon cluster log file level = debug 2023-06-23T01:08:00.008 INFO:teuthology.orchestra.run.smithi073.stdout: debug asserts on shutdown = true 2023-06-23T01:08:00.008 INFO:teuthology.orchestra.run.smithi073.stdout: mon health detail to clog = false 2023-06-23T01:08:00.009 INFO:teuthology.orchestra.run.smithi073.stdout: mon host = "172.21.15.39,172.21.15.73,[v2:172.21.15.73:3301,v1:172.21.15.73:6790]" 2023-06-23T01:08:00.009 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on skipped message = False 2023-06-23T01:08:00.009 INFO:teuthology.orchestra.run.smithi073.stdout: fsid = 314a4b16-8059-4f2c-986c-1624c6de1afb 2023-06-23T01:08:00.009 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.009 INFO:teuthology.orchestra.run.smithi073.stdout:[osd] 2023-06-23T01:08:00.009 INFO:teuthology.orchestra.run.smithi073.stdout: osd journal size = 100 2023-06-23T01:08:00.009 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.009 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub load threshold = 5.0 2023-06-23T01:08:00.010 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub max interval = 600 2023-06-23T01:08:00.010 INFO:teuthology.orchestra.run.smithi073.stdout: osd mclock profile = high_recovery_ops 2023-06-23T01:08:00.010 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.010 INFO:teuthology.orchestra.run.smithi073.stdout: osd recover clone overlap = true 2023-06-23T01:08:00.010 INFO:teuthology.orchestra.run.smithi073.stdout: osd recovery max chunk = 1048576 2023-06-23T01:08:00.010 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.010 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug shutdown = true 2023-06-23T01:08:00.011 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug op order = true 2023-06-23T01:08:00.011 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug verify stray on activate = true 2023-06-23T01:08:00.011 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.011 INFO:teuthology.orchestra.run.smithi073.stdout: osd open classes on start = true 2023-06-23T01:08:00.011 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug pg log writeout = true 2023-06-23T01:08:00.011 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.011 INFO:teuthology.orchestra.run.smithi073.stdout: osd deep scrub update digest min age = 30 2023-06-23T01:08:00.012 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.012 INFO:teuthology.orchestra.run.smithi073.stdout: osd map max advance = 10 2023-06-23T01:08:00.012 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.012 INFO:teuthology.orchestra.run.smithi073.stdout: journal zero on create = true 2023-06-23T01:08:00.012 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.012 INFO:teuthology.orchestra.run.smithi073.stdout: filestore ondisk finisher threads = 3 2023-06-23T01:08:00.012 INFO:teuthology.orchestra.run.smithi073.stdout: filestore apply finisher threads = 3 2023-06-23T01:08:00.012 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.013 INFO:teuthology.orchestra.run.smithi073.stdout: bdev debug aio = true 2023-06-23T01:08:00.013 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug misdirected ops = true 2023-06-23T01:08:00.013 INFO:teuthology.orchestra.run.smithi073.stdout: bdev async discard = True 2023-06-23T01:08:00.013 INFO:teuthology.orchestra.run.smithi073.stdout: bdev enable discard = True 2023-06-23T01:08:00.013 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore allocator = bitmap 2023-06-23T01:08:00.013 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore block size = 96636764160 2023-06-23T01:08:00.013 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore fsck on mount = True 2023-06-23T01:08:00.014 INFO:teuthology.orchestra.run.smithi073.stdout: debug bluefs = 1/20 2023-06-23T01:08:00.014 INFO:teuthology.orchestra.run.smithi073.stdout: debug bluestore = 1/20 2023-06-23T01:08:00.014 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2023-06-23T01:08:00.014 INFO:teuthology.orchestra.run.smithi073.stdout: debug osd = 20 2023-06-23T01:08:00.014 INFO:teuthology.orchestra.run.smithi073.stdout: debug rocksdb = 4/10 2023-06-23T01:08:00.014 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T01:08:00.014 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd full ratio = 0.9 2023-06-23T01:08:00.015 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd nearfull ratio = 0.8 2023-06-23T01:08:00.015 INFO:teuthology.orchestra.run.smithi073.stdout: osd failsafe full ratio = 0.95 2023-06-23T01:08:00.015 INFO:teuthology.orchestra.run.smithi073.stdout: osd objectstore = bluestore 2023-06-23T01:08:00.015 INFO:teuthology.orchestra.run.smithi073.stdout: osd op complaint time = 180 2023-06-23T01:08:00.015 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.015 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr] 2023-06-23T01:08:00.015 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2023-06-23T01:08:00.016 INFO:teuthology.orchestra.run.smithi073.stdout: debug mgr = 20 2023-06-23T01:08:00.016 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2023-06-23T01:08:00.016 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2023-06-23T01:08:00.016 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:08:00.016 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:08:00.016 INFO:teuthology.orchestra.run.smithi073.stdout: mgr/telemetry/nag = false 2023-06-23T01:08:00.016 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.017 INFO:teuthology.orchestra.run.smithi073.stdout:[mon] 2023-06-23T01:08:00.017 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2023-06-23T01:08:00.017 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2023-06-23T01:08:00.017 INFO:teuthology.orchestra.run.smithi073.stdout: debug paxos = 20 2023-06-23T01:08:00.017 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2023-06-23T01:08:00.017 INFO:teuthology.orchestra.run.smithi073.stdout: mon data avail warn = 5 2023-06-23T01:08:00.017 INFO:teuthology.orchestra.run.smithi073.stdout: mon mgr mkfs grace = 240 2023-06-23T01:08:00.017 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:08:00.018 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd reporter subtree level = osd 2023-06-23T01:08:00.018 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd prime pg temp = true 2023-06-23T01:08:00.018 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:08:00.018 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.018 INFO:teuthology.orchestra.run.smithi073.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T01:08:00.018 INFO:teuthology.orchestra.run.smithi073.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T01:08:00.018 INFO:teuthology.orchestra.run.smithi073.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T01:08:00.019 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.019 INFO:teuthology.orchestra.run.smithi073.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T01:08:00.019 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T01:08:00.019 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T01:08:00.019 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.019 INFO:teuthology.orchestra.run.smithi073.stdout: # 1m isn't quite enough 2023-06-23T01:08:00.019 INFO:teuthology.orchestra.run.smithi073.stdout: mon_down_mkfs_grace = 2m 2023-06-23T01:08:00.019 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.020 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_filestore_osds = false 2023-06-23T01:08:00.020 INFO:teuthology.orchestra.run.smithi073.stdout: mon op complaint time = 120 2023-06-23T01:08:00.020 INFO:teuthology.orchestra.run.smithi073.stdout: 2023-06-23T01:08:00.020 INFO:teuthology.orchestra.run.smithi073.stdout:[client] 2023-06-23T01:08:00.020 INFO:teuthology.orchestra.run.smithi073.stdout: rgw cache enabled = true 2023-06-23T01:08:00.020 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable ops log = true 2023-06-23T01:08:00.020 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable usage log = true 2023-06-23T01:08:00.020 INFO:teuthology.orchestra.run.smithi073.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T01:08:00.021 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T01:08:00.021 INFO:teuthology.orchestra.run.smithi073.stdout: client mount timeout = 600 2023-06-23T01:08:00.021 INFO:teuthology.orchestra.run.smithi073.stdout: debug client = 20 2023-06-23T01:08:00.021 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2023-06-23T01:08:00.021 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = "" 2023-06-23T01:08:00.021 INFO:teuthology.orchestra.run.smithi073.stdout: rados mon op timeout = 900 2023-06-23T01:08:00.021 INFO:teuthology.orchestra.run.smithi073.stdout: rados osd op timeout = 900 2023-06-23T01:08:00.022 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.a] 2023-06-23T01:08:00.022 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.b] 2023-06-23T01:08:00.022 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.c] 2023-06-23T01:08:00.022 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.a] 2023-06-23T01:08:00.022 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.c] 2023-06-23T01:08:00.022 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.b] 2023-06-23T01:08:00.022 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.d] 2023-06-23T01:08:00.022 INFO:teuthology.orchestra.run.smithi073.stdout:[mds] 2023-06-23T01:08:00.023 INFO:teuthology.orchestra.run.smithi073.stdout: debug mds = 20 2023-06-23T01:08:00.023 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2023-06-23T01:08:00.023 INFO:teuthology.orchestra.run.smithi073.stdout: mds debug frag = True 2023-06-23T01:08:00.023 INFO:teuthology.orchestra.run.smithi073.stdout: mds debug scatterstat = True 2023-06-23T01:08:00.023 INFO:teuthology.orchestra.run.smithi073.stdout: mds op complaint time = 180 2023-06-23T01:08:00.023 INFO:teuthology.orchestra.run.smithi073.stdout: mds verify scatter = True 2023-06-23T01:08:00.023 INFO:teuthology.orchestra.run.smithi073.stdout: osd op complaint time = 180 2023-06-23T01:08:00.023 INFO:teuthology.orchestra.run.smithi073.stdout: rados mon op timeout = 900 2023-06-23T01:08:00.024 INFO:teuthology.orchestra.run.smithi073.stdout: rados osd op timeout = 900 2023-06-23T01:08:00.024 INFO:tasks.ceph:Creating admin key on mon.a... 2023-06-23T01:08:00.024 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-06-23T01:08:00.097 INFO:tasks.ceph:Copying monmap to all nodes... 2023-06-23T01:08:00.098 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:00.098 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-06-23T01:08:00.114 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:00.115 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-06-23T01:08:00.169 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:08:00.169 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:00.170 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T01:08:00.170 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:08:00.244 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:00.244 DEBUG:teuthology.orchestra.run.smithi039:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:00.298 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:08:00.299 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:00.299 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T01:08:00.300 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:08:00.335 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:00.335 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:00.391 INFO:tasks.ceph:Setting up mon nodes... 2023-06-23T01:08:00.392 INFO:tasks.ceph:Setting up mgr nodes... 2023-06-23T01:08:00.393 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T01:08:00.444 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T01:08:00.447 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T01:08:00.523 INFO:teuthology.orchestra.run.smithi073.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T01:08:00.526 INFO:tasks.ceph:Setting up mds nodes... 2023-06-23T01:08:00.527 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T01:08:00.577 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T01:08:00.579 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-06-23T01:08:00.605 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T01:08:00.696 INFO:teuthology.orchestra.run.smithi039.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T01:08:00.698 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-06-23T01:08:00.724 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T01:08:00.773 INFO:teuthology.orchestra.run.smithi073.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T01:08:00.776 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-06-23T01:08:00.801 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T01:08:00.891 INFO:teuthology.orchestra.run.smithi073.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T01:08:00.894 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-06-23T01:08:00.918 INFO:tasks.ceph_client:Setting up client nodes... 2023-06-23T01:08:00.919 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-06-23T01:08:00.959 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-06-23T01:08:00.974 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-06-23T01:08:01.053 INFO:teuthology.orchestra.run.smithi039.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-06-23T01:08:01.066 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-06-23T01:08:01.105 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-06-23T01:08:01.117 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-06-23T01:08:01.194 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-06-23T01:08:01.206 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-06-23T01:08:01.207 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi039.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T01:08:01.207 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-06-23T01:08:01.233 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:08:01.233 INFO:tasks.ceph:role: osd.0 2023-06-23T01:08:01.233 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:08:01.234 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T01:08:01.325 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:08:01.326 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:08:01.326 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:08:01.326 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 2023-06-23T01:08:01.326 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:08:01.326 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:08:01.326 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:08:01.327 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:08:01.327 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:08:01.327 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:08:02.605 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-06-23T01:08:02.607 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-06-23T01:08:02.607 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-06-23T01:08:02.651 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-06-23T01:08:02.723 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-06-23T01:08:02.791 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:08:02.791 INFO:tasks.ceph:role: osd.1 2023-06-23T01:08:02.792 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:08:02.792 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T01:08:02.883 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:08:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:08:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:08:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 2023-06-23T01:08:02.884 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:08:02.884 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:08:02.885 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:08:02.885 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:08:02.885 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:08:02.885 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:08:04.160 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-06-23T01:08:04.162 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-06-23T01:08:04.162 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-06-23T01:08:04.204 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-06-23T01:08:04.274 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-06-23T01:08:04.341 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:08:04.341 INFO:tasks.ceph:role: osd.2 2023-06-23T01:08:04.341 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:08:04.341 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T01:08:04.436 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:08:04.436 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:08:04.436 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:08:04.437 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 2023-06-23T01:08:04.437 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:08:04.437 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:08:04.437 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:08:04.437 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:08:04.438 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:08:04.438 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:08:05.728 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-06-23T01:08:05.731 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-06-23T01:08:05.732 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-06-23T01:08:05.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-06-23T01:08:05.848 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-06-23T01:08:05.914 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:08:05.915 INFO:tasks.ceph:role: osd.3 2023-06-23T01:08:05.915 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:08:05.915 DEBUG:teuthology.orchestra.run.smithi039:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T01:08:06.007 INFO:teuthology.orchestra.run.smithi039.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:08:06.008 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:08:06.008 INFO:teuthology.orchestra.run.smithi039.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:08:06.008 INFO:teuthology.orchestra.run.smithi039.stdout: = reflink=1 2023-06-23T01:08:06.008 INFO:teuthology.orchestra.run.smithi039.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:08:06.008 INFO:teuthology.orchestra.run.smithi039.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:08:06.009 INFO:teuthology.orchestra.run.smithi039.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:08:06.009 INFO:teuthology.orchestra.run.smithi039.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:08:06.009 INFO:teuthology.orchestra.run.smithi039.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:08:06.009 INFO:teuthology.orchestra.run.smithi039.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:08:07.296 INFO:teuthology.orchestra.run.smithi039.stdout:Discarding blocks...Done. 2023-06-23T01:08:07.297 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi039.front.sepia.ceph.com -o noatime 2023-06-23T01:08:07.298 DEBUG:teuthology.orchestra.run.smithi039:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-06-23T01:08:07.345 DEBUG:teuthology.orchestra.run.smithi039:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-06-23T01:08:07.375 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:07.468 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:07.464+0000 7f0940abc540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-06-23T01:08:07.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:07.464+0000 7f0940abc540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-06-23T01:08:07.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:07.464+0000 7f0940abc540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:08:07.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:07.464+0000 7f0940abc540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:08:07.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:07.464+0000 7f0940abc540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:08:07.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:07.465+0000 7f0940abc540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-06-23T01:08:08.007 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:08.003+0000 7f0940abc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:08.781 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:08.778+0000 7f0940abc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:09.561 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:09.558+0000 7f0940abc540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:10.391 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-06-23T01:08:10.416 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:10.498 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:10.495+0000 7fb2589b5540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-06-23T01:08:10.499 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:10.495+0000 7fb2589b5540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-06-23T01:08:10.499 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:10.495+0000 7fb2589b5540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:08:10.499 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:10.495+0000 7fb2589b5540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:08:10.499 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:10.495+0000 7fb2589b5540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:08:10.500 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:10.496+0000 7fb2589b5540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-06-23T01:08:11.024 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:11.021+0000 7fb2589b5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:11.792 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:11.789+0000 7fb2589b5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:12.554 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:12.551+0000 7fb2589b5540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:13.394 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-06-23T01:08:13.460 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:13.543 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:13.539+0000 7f5546559540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-06-23T01:08:13.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:13.540+0000 7f5546559540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-06-23T01:08:13.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:13.540+0000 7f5546559540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:08:13.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:13.540+0000 7f5546559540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:08:13.544 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:13.540+0000 7f5546559540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:08:13.545 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:13.541+0000 7f5546559540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-06-23T01:08:14.080 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:14.077+0000 7f5546559540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:14.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:14.856+0000 7f5546559540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:15.634 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:15.631+0000 7f5546559540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:16.454 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-06-23T01:08:16.519 DEBUG:teuthology.orchestra.run.smithi039:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:16.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:16.599+0000 7f84e4f09540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-06-23T01:08:16.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:16.599+0000 7f84e4f09540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-06-23T01:08:16.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:16.599+0000 7f84e4f09540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:08:16.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:16.599+0000 7f84e4f09540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:08:16.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:16.599+0000 7f84e4f09540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:08:16.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:16.600+0000 7f84e4f09540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-06-23T01:08:17.136 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:17.133+0000 7f84e4f09540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:17.897 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:17.894+0000 7f84e4f09540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:18.670 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:18.667+0000 7f84e4f09540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:19.510 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-06-23T01:08:19.581 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-06-23T01:08:19.611 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:08:19.612 INFO:tasks.ceph:role: osd.4 2023-06-23T01:08:19.612 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:08:19.612 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T01:08:19.689 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:08:19.689 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:08:19.689 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:08:19.689 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 2023-06-23T01:08:19.689 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:08:19.689 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:08:19.690 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:08:19.690 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:08:19.690 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:08:19.690 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:08:20.114 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2023-06-23T01:08:20.117 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2023-06-23T01:08:20.118 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-06-23T01:08:20.308 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-06-23T01:08:20.345 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-06-23T01:08:20.415 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:08:20.416 INFO:tasks.ceph:role: osd.5 2023-06-23T01:08:20.416 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:08:20.416 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T01:08:20.491 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:08:20.491 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:08:20.492 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:08:20.492 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 2023-06-23T01:08:20.492 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:08:20.492 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:08:20.492 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:08:20.492 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:08:20.493 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:08:20.493 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:08:20.914 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2023-06-23T01:08:20.916 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2023-06-23T01:08:20.916 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-06-23T01:08:21.027 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-06-23T01:08:21.055 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-06-23T01:08:21.120 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:08:21.120 INFO:tasks.ceph:role: osd.6 2023-06-23T01:08:21.120 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:08:21.121 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T01:08:21.192 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:08:21.192 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:08:21.192 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:08:21.192 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 2023-06-23T01:08:21.193 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:08:21.193 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:08:21.193 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:08:21.193 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:08:21.193 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:08:21.193 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:08:21.642 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2023-06-23T01:08:21.644 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2023-06-23T01:08:21.645 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-06-23T01:08:21.802 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-06-23T01:08:21.833 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-06-23T01:08:21.898 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:08:21.898 INFO:tasks.ceph:role: osd.7 2023-06-23T01:08:21.899 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:08:21.899 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T01:08:21.970 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:08:21.970 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:08:21.970 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:08:21.970 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 2023-06-23T01:08:21.971 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:08:21.971 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:08:21.971 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:08:21.971 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:08:21.971 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:08:21.971 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:08:22.417 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2023-06-23T01:08:22.418 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2023-06-23T01:08:22.419 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-06-23T01:08:22.603 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-06-23T01:08:22.636 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:22.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:22.723+0000 7f18864f3540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-06-23T01:08:22.726 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:22.723+0000 7f18864f3540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-06-23T01:08:22.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:22.723+0000 7f18864f3540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:08:22.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:22.723+0000 7f18864f3540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:08:22.727 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:22.723+0000 7f18864f3540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:08:22.728 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:22.724+0000 7f18864f3540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-06-23T01:08:23.249 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:23.246+0000 7f18864f3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:24.017 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:24.014+0000 7f18864f3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:24.781 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:24.778+0000 7f18864f3540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:25.575 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-06-23T01:08:25.602 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:25.685 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:25.681+0000 7f807a28b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-06-23T01:08:25.685 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:25.681+0000 7f807a28b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-06-23T01:08:25.685 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:25.681+0000 7f807a28b540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:08:25.685 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:25.681+0000 7f807a28b540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:08:25.685 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:25.681+0000 7f807a28b540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:08:25.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:25.682+0000 7f807a28b540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-06-23T01:08:26.207 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:26.204+0000 7f807a28b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:26.967 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:26.963+0000 7f807a28b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:27.730 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:27.726+0000 7f807a28b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:27.972 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:08:27.975 DEBUG:teuthology.orchestra.run.smithi073:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:08:28.531 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-06-23T01:08:28.558 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:28.642 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:28.638+0000 7fec4669f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-06-23T01:08:28.642 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:28.638+0000 7fec4669f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-06-23T01:08:28.643 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:28.638+0000 7fec4669f540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:08:28.643 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:28.638+0000 7fec4669f540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:08:28.643 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:28.638+0000 7fec4669f540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:08:28.643 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:28.639+0000 7fec4669f540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-06-23T01:08:29.167 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:29.164+0000 7fec4669f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:29.929 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:29.925+0000 7fec4669f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:30.696 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:30.693+0000 7fec4669f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:31.509 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-06-23T01:08:31.576 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:31.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:31.654+0000 7f7c09e3e540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-06-23T01:08:31.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:31.654+0000 7f7c09e3e540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-06-23T01:08:31.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:31.654+0000 7f7c09e3e540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:08:31.658 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:31.654+0000 7f7c09e3e540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:08:31.659 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:31.654+0000 7f7c09e3e540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:08:31.659 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:31.655+0000 7f7c09e3e540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-06-23T01:08:32.183 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:32.179+0000 7f7c09e3e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:32.944 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:32.941+0000 7f7c09e3e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:33.706 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:33.703+0000 7f7c09e3e540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:08:34.508 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-06-23T01:08:34.574 INFO:tasks.ceph:Reading keys from all nodes... 2023-06-23T01:08:34.575 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:34.575 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-06-23T01:08:34.605 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:34.605 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-06-23T01:08:34.673 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:34.673 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-06-23T01:08:34.741 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:34.741 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-06-23T01:08:34.808 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:34.808 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-06-23T01:08:34.874 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:34.874 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-06-23T01:08:34.940 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:34.940 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-06-23T01:08:35.006 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.006 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-06-23T01:08:35.034 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.034 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-06-23T01:08:35.100 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.101 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-06-23T01:08:35.166 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.166 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-06-23T01:08:35.231 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.231 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-06-23T01:08:35.295 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.296 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-06-23T01:08:35.359 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.359 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-06-23T01:08:35.423 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:35.423 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-06-23T01:08:35.440 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:35.440 DEBUG:teuthology.orchestra.run.smithi039:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-06-23T01:08:35.496 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.496 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-06-23T01:08:35.511 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:35.511 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-06-23T01:08:35.565 INFO:tasks.ceph:Adding keys to all mons... 2023-06-23T01:08:35.565 DEBUG:teuthology.orchestra.run.smithi039:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T01:08:35.567 DEBUG:teuthology.orchestra.run.smithi073:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T01:08:35.608 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.y] 2023-06-23T01:08:35.608 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBw8JRkl71jGhAAud5Ohexz90GorO0WOMEcGg== 2023-06-23T01:08:35.608 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.a] 2023-06-23T01:08:35.608 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBw8JRkTPpQIhAAx3znAiGN5cKvQSWWYybbIA== 2023-06-23T01:08:35.608 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.c] 2023-06-23T01:08:35.608 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBw8JRkjWlaKRAAvC2mKUvyGjZB9FLoqd0cnA== 2023-06-23T01:08:35.609 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.0] 2023-06-23T01:08:35.609 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB38JRkjiXKGxAAZL2AbQ5lzV5XMMXTWuMuoQ== 2023-06-23T01:08:35.609 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.1] 2023-06-23T01:08:35.609 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB68JRkpBSPHRAAd19lKbiaiy6u4eq6Iu3iiQ== 2023-06-23T01:08:35.609 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.2] 2023-06-23T01:08:35.609 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQB98JRk1KY8IBAAr56i06y/oZvteRfJJotM8Q== 2023-06-23T01:08:35.609 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.3] 2023-06-23T01:08:35.610 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCA8JRkL6jHIxAAmjtZMPLoBt/3qks0bH3K0Q== 2023-06-23T01:08:35.610 INFO:teuthology.orchestra.run.smithi039.stdout:[mgr.x] 2023-06-23T01:08:35.610 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBw8JRkotsOHxAASfBH4j2xM3By/1q3vPHXYg== 2023-06-23T01:08:35.610 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.b] 2023-06-23T01:08:35.610 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBw8JRkftH5LRAAgyAJ7kJCLLCEh+8JWdY80w== 2023-06-23T01:08:35.610 INFO:teuthology.orchestra.run.smithi039.stdout:[mds.d] 2023-06-23T01:08:35.610 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBw8JRkEej/NBAAxqoHApL2jLNtOQer9Jjv9Q== 2023-06-23T01:08:35.610 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.4] 2023-06-23T01:08:35.611 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCG8JRkxMArKxAA2bZV6QDE7XcY9FLG/gwHeQ== 2023-06-23T01:08:35.611 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.5] 2023-06-23T01:08:35.611 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCJ8JRkPc6wKBAADgKbEuS76UwQOrBe8w6Lmw== 2023-06-23T01:08:35.611 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.6] 2023-06-23T01:08:35.611 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCM8JRkF7wiJhAA4TdLQF4l8vd07dIG1Q53sg== 2023-06-23T01:08:35.611 INFO:teuthology.orchestra.run.smithi039.stdout:[osd.7] 2023-06-23T01:08:35.611 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQCP8JRknDoYJxAAam3KvDIfXI5zp9e1EyEh3A== 2023-06-23T01:08:35.611 INFO:teuthology.orchestra.run.smithi039.stdout:[client.0] 2023-06-23T01:08:35.612 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBw8JRkiAMMORAAIYhNG3q7AQKLX47D1vhQTA== 2023-06-23T01:08:35.612 INFO:teuthology.orchestra.run.smithi039.stdout:[client.2] 2023-06-23T01:08:35.612 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBx8JRk8TYKAxAA365bDeDG/uTRzjrKEagFDg== 2023-06-23T01:08:35.612 INFO:teuthology.orchestra.run.smithi039.stdout:[client.1] 2023-06-23T01:08:35.612 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBx8JRknswXBhAARuGLcdPyAkIcGuU8YoB7Dw== 2023-06-23T01:08:35.612 INFO:teuthology.orchestra.run.smithi039.stdout:[client.3] 2023-06-23T01:08:35.612 INFO:teuthology.orchestra.run.smithi039.stdout: key = AQBx8JRkDjt1CxAA+qGjbg79NclRfDQELprCOQ== 2023-06-23T01:08:35.627 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr.y] 2023-06-23T01:08:35.628 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBw8JRkl71jGhAAud5Ohexz90GorO0WOMEcGg== 2023-06-23T01:08:35.628 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.a] 2023-06-23T01:08:35.628 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBw8JRkTPpQIhAAx3znAiGN5cKvQSWWYybbIA== 2023-06-23T01:08:35.628 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.c] 2023-06-23T01:08:35.628 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBw8JRkjWlaKRAAvC2mKUvyGjZB9FLoqd0cnA== 2023-06-23T01:08:35.628 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.0] 2023-06-23T01:08:35.628 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQB38JRkjiXKGxAAZL2AbQ5lzV5XMMXTWuMuoQ== 2023-06-23T01:08:35.629 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.1] 2023-06-23T01:08:35.629 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQB68JRkpBSPHRAAd19lKbiaiy6u4eq6Iu3iiQ== 2023-06-23T01:08:35.629 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.2] 2023-06-23T01:08:35.629 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQB98JRk1KY8IBAAr56i06y/oZvteRfJJotM8Q== 2023-06-23T01:08:35.629 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.3] 2023-06-23T01:08:35.629 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQCA8JRkL6jHIxAAmjtZMPLoBt/3qks0bH3K0Q== 2023-06-23T01:08:35.629 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr.x] 2023-06-23T01:08:35.629 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBw8JRkotsOHxAASfBH4j2xM3By/1q3vPHXYg== 2023-06-23T01:08:35.630 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.b] 2023-06-23T01:08:35.630 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBw8JRkftH5LRAAgyAJ7kJCLLCEh+8JWdY80w== 2023-06-23T01:08:35.630 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.d] 2023-06-23T01:08:35.630 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBw8JRkEej/NBAAxqoHApL2jLNtOQer9Jjv9Q== 2023-06-23T01:08:35.630 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.4] 2023-06-23T01:08:35.630 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQCG8JRkxMArKxAA2bZV6QDE7XcY9FLG/gwHeQ== 2023-06-23T01:08:35.630 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.5] 2023-06-23T01:08:35.630 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQCJ8JRkPc6wKBAADgKbEuS76UwQOrBe8w6Lmw== 2023-06-23T01:08:35.631 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.6] 2023-06-23T01:08:35.631 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQCM8JRkF7wiJhAA4TdLQF4l8vd07dIG1Q53sg== 2023-06-23T01:08:35.631 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.7] 2023-06-23T01:08:35.631 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQCP8JRknDoYJxAAam3KvDIfXI5zp9e1EyEh3A== 2023-06-23T01:08:35.631 INFO:teuthology.orchestra.run.smithi073.stdout:[client.0] 2023-06-23T01:08:35.631 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBw8JRkiAMMORAAIYhNG3q7AQKLX47D1vhQTA== 2023-06-23T01:08:35.631 INFO:teuthology.orchestra.run.smithi073.stdout:[client.2] 2023-06-23T01:08:35.632 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBx8JRk8TYKAxAA365bDeDG/uTRzjrKEagFDg== 2023-06-23T01:08:35.632 INFO:teuthology.orchestra.run.smithi073.stdout:[client.1] 2023-06-23T01:08:35.632 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBx8JRknswXBhAARuGLcdPyAkIcGuU8YoB7Dw== 2023-06-23T01:08:35.632 INFO:teuthology.orchestra.run.smithi073.stdout:[client.3] 2023-06-23T01:08:35.632 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQBx8JRkDjt1CxAA+qGjbg79NclRfDQELprCOQ== 2023-06-23T01:08:35.633 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:08:35.656 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:08:35.721 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:08:35.723 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:08:35.775 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:08:35.778 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:08:35.829 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:35.831 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:35.878 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:35.881 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:35.928 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:35.930 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:35.977 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:35.979 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.027 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:08:36.029 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:08:36.077 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:08:36.079 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:08:36.129 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:08:36.131 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:08:36.181 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.184 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.230 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.233 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.279 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.281 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.328 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.330 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:08:36.377 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:08:36.379 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:08:36.429 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:08:36.431 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:08:36.481 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:08:36.483 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:08:36.533 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:08:36.535 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:08:36.585 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-06-23T01:08:36.585 DEBUG:teuthology.orchestra.run.smithi039:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-06-23T01:08:36.610 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:08:36.705 DEBUG:teuthology.orchestra.run.smithi039:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-06-23T01:08:36.730 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-06-23T01:08:36.754 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:08:36.846 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-06-23T01:08:36.870 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-06-23T01:08:36.934 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:08:37.025 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-06-23T01:08:37.049 DEBUG:teuthology.orchestra.run.smithi039:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:37.051 DEBUG:teuthology.orchestra.run.smithi073:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:08:37.103 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-06-23T01:08:37.104 INFO:tasks.ceph.mon.a:Restarting daemon 2023-06-23T01:08:37.104 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-06-23T01:08:37.106 INFO:tasks.ceph.mon.a:Started 2023-06-23T01:08:37.107 INFO:tasks.ceph.mon.b:Restarting daemon 2023-06-23T01:08:37.107 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-06-23T01:08:37.146 INFO:tasks.ceph.mon.b:Started 2023-06-23T01:08:37.147 INFO:tasks.ceph.mon.c:Restarting daemon 2023-06-23T01:08:37.147 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-06-23T01:08:37.149 INFO:tasks.ceph.mon.c:Started 2023-06-23T01:08:37.149 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-06-23T01:08:37.149 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-06-23T01:08:37.149 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-06-23T01:08:37.151 INFO:tasks.ceph.mgr.y:Started 2023-06-23T01:08:37.152 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-06-23T01:08:37.152 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-06-23T01:08:37.153 INFO:tasks.ceph.mgr.x:Started 2023-06-23T01:08:37.154 INFO:tasks.ceph:Setting crush tunables to default 2023-06-23T01:08:37.154 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd crush tunables default 2023-06-23T01:08:37.273 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.270+0000 7f549eb1d700 1 Processor -- start 2023-06-23T01:08:37.274 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.270+0000 7f549eb1d700 1 -- start start 2023-06-23T01:08:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.271+0000 7f549eb1d700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 0x7f54980cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.271+0000 7f549eb1d700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 0x7f54980c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.271+0000 7f549eb1d700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54980c88e0 con 0x7f5498142f50 2023-06-23T01:08:37.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.271+0000 7f549eb1d700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5498143320 con 0x7f54980cb230 2023-06-23T01:08:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.271+0000 7f549eb1d700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54980cb670 con 0x7f54980c7760 2023-06-23T01:08:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.271+0000 7f549c8b9700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f5498142f50 0x7f5498068750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54872/0 (socket says 172.21.15.39:54872) 2023-06-23T01:08:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.271+0000 7f549c8b9700 1 -- 172.21.15.39:0/2999713505 learned_addr learned my addr 172.21.15.39:0/2999713505 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f5497fff700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 0x7f54980cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f549d0ba700 1 --2- 172.21.15.39:0/2999713505 >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 0x7f54980c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f5497fff700 1 -- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 msgr2=0x7f54980cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-23T01:08:37.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f5497fff700 1 -- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 msgr2=0x7f54980cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:08:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f5497fff700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 0x7f54980cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:08:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f549d0ba700 1 -- 172.21.15.39:0/2999713505 >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 msgr2=0x7f54980c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-23T01:08:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f549d0ba700 1 -- 172.21.15.39:0/2999713505 >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 msgr2=0x7f54980c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:08:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f549d0ba700 1 --2- 172.21.15.39:0/2999713505 >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 0x7f54980c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:08:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f5497fff700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 0x7f54980cacf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:08:37.277 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.272+0000 7f549d0ba700 1 --2- 172.21.15.39:0/2999713505 >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 0x7f54980c8d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:08:37.475 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.472+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937086869 0 0) 0x7f54980c88e0 con 0x7f5498142f50 2023-06-23T01:08:37.475 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.472+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5484003570 con 0x7f5498142f50 2023-06-23T01:08:37.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f5497fff700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 0x7f54980cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f549d0ba700 1 --2- 172.21.15.39:0/2999713505 >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 0x7f54980c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 445249637 0 0) 0x7f5484003570 con 0x7f5498142f50 2023-06-23T01:08:37.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 msgr2=0x7f54980cacf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:37.477 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 0x7f54980cacf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 msgr2=0x7f54980c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 --2- 172.21.15.39:0/2999713505 >> v2:172.21.15.73:3300/0 conn(0x7f54980cb230 0x7f54980c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980c7cd0 con 0x7f5498142f50 2023-06-23T01:08:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 979813403 0 0) 0x7f5490001470 con 0x7f5498142f50 2023-06-23T01:08:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 >> v1:172.21.15.39:6789/0 conn(0x7f5498142f50 legacy=0x7f5498068750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5484003a60 0x7f5484003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:37.478 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5484004810 0x7f5484004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54840051c0 0x7f548400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5498143320 con 0x7f5484004810 2023-06-23T01:08:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54980cb670 con 0x7f5484003a60 2023-06-23T01:08:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548400c050 con 0x7f54840051c0 2023-06-23T01:08:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f5497fff700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54980c7760 0x7f54980cacf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:08:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f549d0ba700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54840051c0 0x7f548400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f549c8b9700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5484003a60 0x7f5484003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.479 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.473+0000 7f5497fff700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5484004810 0x7f5484004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f549d0ba700 1 -- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5484003a60 msgr2=0x7f5484003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f549d0ba700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5484003a60 0x7f5484003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f549d0ba700 1 -- 172.21.15.39:0/2999713505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5484004810 msgr2=0x7f5484004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f549d0ba700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f5484004810 0x7f5484004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f549d0ba700 1 -- 172.21.15.39:0/2999713505 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484003570 con 0x7f54840051c0 2023-06-23T01:08:37.480 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f549d0ba700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54840051c0 0x7f548400ba60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f548000dc00 tx=0x7f5480008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85e3e215d092af98 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:37.482 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548000d570 con 0x7f54840051c0 2023-06-23T01:08:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548000f720 con 0x7f54840051c0 2023-06-23T01:08:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.474+0000 7f54977fe700 1 -- 172.21.15.39:0/2999713505 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548000f8d0 con 0x7f54840051c0 2023-06-23T01:08:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.475+0000 7f549eb1d700 1 -- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54840051c0 msgr2=0x7f548400ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.475+0000 7f549eb1d700 1 --2- 172.21.15.39:0/2999713505 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f54840051c0 0x7f548400ba60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f548000dc00 tx=0x7f5480008040 comp rx=0 tx=0).stop 2023-06-23T01:08:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.475+0000 7f549d0ba700 1 -- 172.21.15.39:0/2999713505 reap_dead start 2023-06-23T01:08:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.475+0000 7f549eb1d700 1 -- 172.21.15.39:0/2999713505 shutdown_connections 2023-06-23T01:08:37.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.475+0000 7f549eb1d700 1 -- 172.21.15.39:0/2999713505 >> 172.21.15.39:0/2999713505 conn(0x7f549806b250 msgr2=0x7f549806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.476+0000 7f549eb1d700 1 -- 172.21.15.39:0/2999713505 shutdown_connections 2023-06-23T01:08:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.476+0000 7f549eb1d700 1 -- 172.21.15.39:0/2999713505 wait complete. 2023-06-23T01:08:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.477+0000 7f549eb1d700 1 Processor -- start 2023-06-23T01:08:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.477+0000 7f549eb1d700 1 -- start start 2023-06-23T01:08:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549eb1d700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f549813aeb0 0x7f5498140ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549eb1d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f549813b280 0x7f54980c6ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549eb1d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f549813bfe0 0x7f549816cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:37.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549eb1d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54981416c0 con 0x7f549813bfe0 2023-06-23T01:08:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549eb1d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54980c41d0 con 0x7f549813b280 2023-06-23T01:08:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549eb1d700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54980c7020 con 0x7f549813aeb0 2023-06-23T01:08:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549c8b9700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f549813aeb0 0x7f5498140ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549d0ba700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f549813bfe0 0x7f549816cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549c8b9700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f549813aeb0 0x7f5498140ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.39:39536/0 (socket says 172.21.15.39:39536) 2023-06-23T01:08:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549c8b9700 1 -- 172.21.15.39:0/876082164 learned_addr learned my addr 172.21.15.39:0/876082164 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:37.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f5497fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f549813b280 0x7f54980c6ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549c8b9700 1 -- 172.21.15.39:0/876082164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f549813b280 msgr2=0x7f54980c6ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549c8b9700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f549813b280 0x7f54980c6ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549c8b9700 1 -- 172.21.15.39:0/876082164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f549813bfe0 msgr2=0x7f549816cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549c8b9700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f549813bfe0 0x7f549816cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549c8b9700 1 -- 172.21.15.39:0/876082164 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5498141100 con 0x7f549813aeb0 2023-06-23T01:08:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f5497fff700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f549813b280 0x7f54980c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.478+0000 7f549d0ba700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f549813bfe0 0x7f549816cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:37.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.479+0000 7f549c8b9700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f549813aeb0 0x7f5498140ad0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54900015a0 tx=0x7f5490001e90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:37.487 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.480+0000 7f5494ff9700 1 -- 172.21.15.39:0/876082164 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5490018030 con 0x7f549813aeb0 2023-06-23T01:08:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.480+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54981413e0 con 0x7f549813aeb0 2023-06-23T01:08:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.480+0000 7f5494ff9700 1 -- 172.21.15.39:0/876082164 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54900077c0 con 0x7f549813aeb0 2023-06-23T01:08:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.480+0000 7f5494ff9700 1 -- 172.21.15.39:0/876082164 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549000fa70 con 0x7f549813aeb0 2023-06-23T01:08:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.480+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54980c46c0 con 0x7f549813aeb0 2023-06-23T01:08:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.480+0000 7f5494ff9700 1 -- 172.21.15.39:0/876082164 <== mon.2 v2:172.21.15.73:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f549001b070 con 0x7f549813aeb0 2023-06-23T01:08:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.480+0000 7f5494ff9700 1 -- 172.21.15.39:0/876082164 <== mon.2 v2:172.21.15.73:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5490019070 con 0x7f549813aeb0 2023-06-23T01:08:37.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.481+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54980c48e0 con 0x7f549813aeb0 2023-06-23T01:08:37.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.489+0000 7f5494ff9700 1 -- 172.21.15.39:0/876082164 <== mon.2 v2:172.21.15.73:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f549001e470 con 0x7f549813aeb0 2023-06-23T01:08:37.567 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.564+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f549804c1b0 con 0x7f549813aeb0 2023-06-23T01:08:37.579 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.576+0000 7f5494ff9700 1 -- 172.21.15.39:0/876082164 <== mon.2 v2:172.21.15.73:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f549001e610 con 0x7f549813aeb0 2023-06-23T01:08:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.577+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f549813aeb0 msgr2=0x7f5498140ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.577+0000 7f549eb1d700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f549813aeb0 0x7f5498140ad0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f54900015a0 tx=0x7f5490001e90 comp rx=0 tx=0).stop 2023-06-23T01:08:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.577+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 shutdown_connections 2023-06-23T01:08:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.577+0000 7f549eb1d700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f549813bfe0 0x7f549816cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.577+0000 7f549eb1d700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f549813aeb0 0x7f5498140ad0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.581 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.577+0000 7f549eb1d700 1 --2- 172.21.15.39:0/876082164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f549813b280 0x7f54980c6ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.577+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 >> 172.21.15.39:0/876082164 conn(0x7f549806b250 msgr2=0x7f54980c2160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.578+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 shutdown_connections 2023-06-23T01:08:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:37.578+0000 7f549eb1d700 1 -- 172.21.15.39:0/876082164 wait complete. 2023-06-23T01:08:37.582 INFO:teuthology.orchestra.run.smithi039.stderr:adjusted tunables profile to default 2023-06-23T01:08:37.593 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-06-23T01:08:37.593 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:37.594 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-06-23T01:08:37.618 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:37.619 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-06-23T01:08:37.683 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:37.684 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-06-23T01:08:37.748 DEBUG:teuthology.orchestra.run.smithi039:> set -ex 2023-06-23T01:08:37.749 DEBUG:teuthology.orchestra.run.smithi039:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-06-23T01:08:37.815 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:37.815 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-06-23T01:08:37.839 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:37.840 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-06-23T01:08:37.904 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:37.905 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-06-23T01:08:37.968 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2023-06-23T01:08:37.968 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-06-23T01:08:38.032 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new a33e557d-f70e-4a1c-a10e-bc7f351dee53 0 2023-06-23T01:08:38.188 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.185+0000 7f245f59e700 1 Processor -- start 2023-06-23T01:08:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.185+0000 7f245f59e700 1 -- start start 2023-06-23T01:08:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245f59e700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f2460067b60 0x7f246015a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245f59e700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f246015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245f59e700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24600ce670 con 0x7f2460156ca0 2023-06-23T01:08:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245f59e700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246015de90 con 0x7f2460067b60 2023-06-23T01:08:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245f59e700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246015d3e0 con 0x7f2460157070 2023-06-23T01:08:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245dd9b700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f2460067b60 0x7f246015a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245e59c700 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f2460156ca0 0x7f2460067450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:47496/0 (socket says 172.21.15.73:47496) 2023-06-23T01:08:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245e59c700 1 -- 172.21.15.73:0/2531166593 learned_addr learned my addr 172.21.15.73:0/2531166593 (peer_addr_for_me v1:172.21.15.73:0/0) 2023-06-23T01:08:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245dd9b700 1 --2- 172.21.15.73:0/2531166593 >> v2:172.21.15.73:3300/0 conn(0x7f2460067b60 0x7f246015a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53264/0 (socket says 172.21.15.73:53264) 2023-06-23T01:08:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245ed9d700 1 --2- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f246015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245dd9b700 1 -- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 msgr2=0x7f246015cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245dd9b700 1 --2- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f246015cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.186+0000 7f245dd9b700 1 -- 172.21.15.73:0/2531166593 >> v1:172.21.15.73:6789/0 conn(0x7f2460156ca0 legacy=0x7f2460067450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245dd9b700 1 -- 172.21.15.73:0/2531166593 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246015d5f0 con 0x7f2460067b60 2023-06-23T01:08:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245d59a700 1 -- 172.21.15.73:0/2531166593 <== mon.1 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131179054 0 0) 0x7f24600ce670 con 0x7f2460156ca0 2023-06-23T01:08:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245ed9d700 1 --2- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f246015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245dd9b700 1 --2- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 0x7f246015a3d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f24500098e0 tx=0x7f24500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e366c49d526f519 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245d59a700 1 -- 172.21.15.73:0/2531166593 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245000b640 con 0x7f2460067b60 2023-06-23T01:08:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245d59a700 1 -- 172.21.15.73:0/2531166593 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245000b7a0 con 0x7f2460067b60 2023-06-23T01:08:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245d59a700 1 -- 172.21.15.73:0/2531166593 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245000bb40 con 0x7f2460067b60 2023-06-23T01:08:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245f59e700 1 -- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 msgr2=0x7f246015a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245f59e700 1 --2- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 0x7f246015a3d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f24500098e0 tx=0x7f24500091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245f59e700 1 -- 172.21.15.73:0/2531166593 shutdown_connections 2023-06-23T01:08:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245f59e700 1 --2- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f246015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245f59e700 1 --2- 172.21.15.73:0/2531166593 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 0x7f246015a3d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245f59e700 1 -- 172.21.15.73:0/2531166593 >> 172.21.15.73:0/2531166593 conn(0x7f24600d0240 msgr2=0x7f24600d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245f59e700 1 -- 172.21.15.73:0/2531166593 shutdown_connections 2023-06-23T01:08:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.187+0000 7f245f59e700 1 -- 172.21.15.73:0/2531166593 wait complete. 2023-06-23T01:08:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245f59e700 1 Processor -- start 2023-06-23T01:08:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245f59e700 1 -- start start 2023-06-23T01:08:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245f59e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 0x7f24600b57d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245f59e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2460156ca0 0x7f2460151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245f59e700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f24600b2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245f59e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2460153b10 con 0x7f2460156ca0 2023-06-23T01:08:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245f59e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2460152440 con 0x7f2460067b60 2023-06-23T01:08:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245f59e700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2460152c20 con 0x7f2460157070 2023-06-23T01:08:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245e59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 0x7f24600b57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245ed9d700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f24600b2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245ed9d700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f24600b2fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.73:43540/0 (socket says 172.21.15.73:43540) 2023-06-23T01:08:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245ed9d700 1 -- 172.21.15.73:0/3220017736 learned_addr learned my addr 172.21.15.73:0/3220017736 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245e59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 0x7f24600b57d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53274/0 (socket says 172.21.15.73:53274) 2023-06-23T01:08:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.189+0000 7f245dd9b700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2460156ca0 0x7f2460151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245ed9d700 1 -- 172.21.15.73:0/3220017736 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 msgr2=0x7f24600b57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245ed9d700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 0x7f24600b57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245ed9d700 1 -- 172.21.15.73:0/3220017736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2460156ca0 msgr2=0x7f2460151890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245ed9d700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2460156ca0 0x7f2460151890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245ed9d700 1 -- 172.21.15.73:0/3220017736 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2458009a20 con 0x7f2460157070 2023-06-23T01:08:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245dd9b700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2460156ca0 0x7f2460151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:08:38.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245ed9d700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f24600b2fe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2458009ef0 tx=0x7f245800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:38.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f244e7fc700 1 -- 172.21.15.73:0/3220017736 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f245800cc20 con 0x7f2460157070 2023-06-23T01:08:38.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f244e7fc700 1 -- 172.21.15.73:0/3220017736 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f245800f040 con 0x7f2460157070 2023-06-23T01:08:38.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2450009420 con 0x7f2460157070 2023-06-23T01:08:38.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f244e7fc700 1 -- 172.21.15.73:0/3220017736 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2458014480 con 0x7f2460157070 2023-06-23T01:08:38.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24601531e0 con 0x7f2460157070 2023-06-23T01:08:38.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f244e7fc700 1 -- 172.21.15.73:0/3220017736 <== mon.2 v2:172.21.15.73:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f24580147f0 con 0x7f2460157070 2023-06-23T01:08:38.200 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.190+0000 7f244e7fc700 1 -- 172.21.15.73:0/3220017736 <== mon.2 v2:172.21.15.73:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f245801da50 con 0x7f2460157070 2023-06-23T01:08:38.201 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.191+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24600d1010 con 0x7f2460157070 2023-06-23T01:08:38.201 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.193+0000 7f244e7fc700 1 -- 172.21.15.73:0/3220017736 <== mon.2 v2:172.21.15.73:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f245803b6e0 con 0x7f2460157070 2023-06-23T01:08:38.278 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.275+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a33e557d-f70e-4a1c-a10e-bc7f351dee53", "id": 0} v 0) v1 -- 0x7f2460153c80 con 0x7f2460157070 2023-06-23T01:08:38.294 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.290+0000 7f244e7fc700 1 -- 172.21.15.73:0/3220017736 <== mon.2 v2:172.21.15.73:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a33e557d-f70e-4a1c-a10e-bc7f351dee53", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f245802a070 con 0x7f2460157070 2023-06-23T01:08:38.294 INFO:teuthology.orchestra.run.smithi073.stdout:0 2023-06-23T01:08:38.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 msgr2=0x7f24600b2fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f24600b2fe0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f2458009ef0 tx=0x7f245800b040 comp rx=0 tx=0).stop 2023-06-23T01:08:38.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 shutdown_connections 2023-06-23T01:08:38.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2460157070 0x7f24600b2fe0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.295 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2460067b60 0x7f24600b57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 --2- 172.21.15.73:0/3220017736 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2460156ca0 0x7f2460151890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 >> 172.21.15.73:0/3220017736 conn(0x7f24600d0240 msgr2=0x7f246015cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:38.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 shutdown_connections 2023-06-23T01:08:38.296 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.291+0000 7f245f59e700 1 -- 172.21.15.73:0/3220017736 wait complete. 2023-06-23T01:08:38.306 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new 757696fc-3c60-4f7f-822d-a8ff1e89534d 1 2023-06-23T01:08:38.421 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.418+0000 7f63baa55700 1 Processor -- start 2023-06-23T01:08:38.421 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.418+0000 7f63baa55700 1 -- start start 2023-06-23T01:08:38.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63baa55700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f63b40b2af0 0x7f63b40cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63baa55700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b415c0e0 con 0x7f63b415af60 2023-06-23T01:08:38.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63baa55700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63b415b4d0 con 0x7f63b40b2720 2023-06-23T01:08:38.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63baa55700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63b415b7b0 con 0x7f63b40b2af0 2023-06-23T01:08:38.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b37fe700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f63b40b2af0 0x7f63b40cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b37fe700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f63b40b2af0 0x7f63b40cd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53284/0 (socket says 172.21.15.73:53284) 2023-06-23T01:08:38.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b37fe700 1 -- 172.21.15.73:0/861574972 learned_addr learned my addr 172.21.15.73:0/861574972 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:38.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b3fff700 1 --1- 172.21.15.73:0/861574972 >> v1:172.21.15.73:6789/0 conn(0x7f63b40b2720 0x7f63b41579d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.73:47506/0 (socket says 172.21.15.73:47506) 2023-06-23T01:08:38.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b37fe700 1 -- 172.21.15.73:0/861574972 >> v1:172.21.15.73:6789/0 conn(0x7f63b40b2720 legacy=0x7f63b41579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b37fe700 1 -- 172.21.15.73:0/861574972 >> v1:172.21.15.39:6789/0 conn(0x7f63b415af60 legacy=0x7f63b415d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b37fe700 1 -- 172.21.15.73:0/861574972 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b415ba90 con 0x7f63b40b2af0 2023-06-23T01:08:38.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b37fe700 1 --2- 172.21.15.73:0/861574972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2af0 0x7f63b40cd6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f63a40099b0 tx=0x7f63a4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f851344f55419e54 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:38.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b2ffd700 1 -- 172.21.15.73:0/861574972 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a400b860 con 0x7f63b40b2af0 2023-06-23T01:08:38.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b2ffd700 1 -- 172.21.15.73:0/861574972 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a400b9c0 con 0x7f63b40b2af0 2023-06-23T01:08:38.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.419+0000 7f63b2ffd700 1 -- 172.21.15.73:0/861574972 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a400bd60 con 0x7f63b40b2af0 2023-06-23T01:08:38.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.420+0000 7f63baa55700 1 -- 172.21.15.73:0/861574972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2af0 msgr2=0x7f63b40cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.420+0000 7f63baa55700 1 --2- 172.21.15.73:0/861574972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2af0 0x7f63b40cd6c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f63a40099b0 tx=0x7f63a4009250 comp rx=0 tx=0).stop 2023-06-23T01:08:38.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.420+0000 7f63baa55700 1 -- 172.21.15.73:0/861574972 shutdown_connections 2023-06-23T01:08:38.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.420+0000 7f63baa55700 1 --2- 172.21.15.73:0/861574972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2af0 0x7f63b40cd6c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.420+0000 7f63baa55700 1 -- 172.21.15.73:0/861574972 >> 172.21.15.73:0/861574972 conn(0x7f63b4067450 msgr2=0x7f63b4067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:38.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.420+0000 7f63baa55700 1 -- 172.21.15.73:0/861574972 shutdown_connections 2023-06-23T01:08:38.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.420+0000 7f63baa55700 1 -- 172.21.15.73:0/861574972 wait complete. 2023-06-23T01:08:38.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.421+0000 7f63baa55700 1 Processor -- start 2023-06-23T01:08:38.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.421+0000 7f63baa55700 1 -- start start 2023-06-23T01:08:38.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63baa55700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2720 0x7f63b414f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63baa55700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63b40b2af0 0x7f63b40b5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.428 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63baa55700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63b415af60 0x7f63b40ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.429 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63baa55700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40b3950 con 0x7f63b40b2af0 2023-06-23T01:08:38.429 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63baa55700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b40b56a0 con 0x7f63b40b2720 2023-06-23T01:08:38.429 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63baa55700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63b414f710 con 0x7f63b415af60 2023-06-23T01:08:38.429 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b3fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2720 0x7f63b414f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.429 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b3fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2720 0x7f63b414f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53296/0 (socket says 172.21.15.73:53296) 2023-06-23T01:08:38.429 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b3fff700 1 -- 172.21.15.73:0/99687838 learned_addr learned my addr 172.21.15.73:0/99687838 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:38.429 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b8ff2700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63b415af60 0x7f63b40ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.429 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b37fe700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63b40b2af0 0x7f63b40b5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.430 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b3fff700 1 -- 172.21.15.73:0/99687838 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63b415af60 msgr2=0x7f63b40ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.430 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b3fff700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63b415af60 0x7f63b40ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.430 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b3fff700 1 -- 172.21.15.73:0/99687838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63b40b2af0 msgr2=0x7f63b40b5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.430 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b3fff700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63b40b2af0 0x7f63b40b5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.430 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.422+0000 7f63b3fff700 1 -- 172.21.15.73:0/99687838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a40094f0 con 0x7f63b40b2720 2023-06-23T01:08:38.430 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.423+0000 7f63b3fff700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2720 0x7f63b414f1d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f63a800c7c0 tx=0x7f63a8007f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:38.430 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.423+0000 7f639bfff700 1 -- 172.21.15.73:0/99687838 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a8005a70 con 0x7f63b40b2720 2023-06-23T01:08:38.431 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.423+0000 7f639bfff700 1 -- 172.21.15.73:0/99687838 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63a8005bd0 con 0x7f63b40b2720 2023-06-23T01:08:38.431 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.423+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b40b6070 con 0x7f63b40b2720 2023-06-23T01:08:38.431 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.423+0000 7f639bfff700 1 -- 172.21.15.73:0/99687838 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63a80070d0 con 0x7f63b40b2720 2023-06-23T01:08:38.431 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.423+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b40b5050 con 0x7f63b40b2720 2023-06-23T01:08:38.431 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.423+0000 7f639bfff700 1 -- 172.21.15.73:0/99687838 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f63a801c440 con 0x7f63b40b2720 2023-06-23T01:08:38.431 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.423+0000 7f639bfff700 1 -- 172.21.15.73:0/99687838 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f63a80127c0 con 0x7f63b40b2720 2023-06-23T01:08:38.431 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.424+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b404c1b0 con 0x7f63b40b2720 2023-06-23T01:08:38.432 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.426+0000 7f639bfff700 1 -- 172.21.15.73:0/99687838 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f63a8017070 con 0x7f63b40b2720 2023-06-23T01:08:38.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.509+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd new", "uuid": "757696fc-3c60-4f7f-822d-a8ff1e89534d", "id": 1} v 0) v1 -- 0x7f63b415bd70 con 0x7f63b40b2720 2023-06-23T01:08:38.515 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.512+0000 7f639bfff700 1 -- 172.21.15.73:0/99687838 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "757696fc-3c60-4f7f-822d-a8ff1e89534d", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f63a80076c0 con 0x7f63b40b2720 2023-06-23T01:08:38.515 INFO:teuthology.orchestra.run.smithi073.stdout:1 2023-06-23T01:08:38.516 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2720 msgr2=0x7f63b414f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.516 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2720 0x7f63b414f1d0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f63a800c7c0 tx=0x7f63a8007f70 comp rx=0 tx=0).stop 2023-06-23T01:08:38.516 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 shutdown_connections 2023-06-23T01:08:38.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63b415af60 0x7f63b40ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63b40b2720 0x7f63b414f1d0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 --2- 172.21.15.73:0/99687838 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63b40b2af0 0x7f63b40b5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 >> 172.21.15.73:0/99687838 conn(0x7f63b4067450 msgr2=0x7f63b4155fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:38.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 shutdown_connections 2023-06-23T01:08:38.517 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.513+0000 7f63baa55700 1 -- 172.21.15.73:0/99687838 wait complete. 2023-06-23T01:08:38.528 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new cffcd6d2-b414-40b0-8967-26421b058b3e 2 2023-06-23T01:08:38.644 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.641+0000 7ff16ec43700 1 Processor -- start 2023-06-23T01:08:38.645 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.641+0000 7ff16ec43700 1 -- start start 2023-06-23T01:08:38.647 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff16ec43700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7ff1680c7e70 0x7ff16806eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.647 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff16ec43700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff1680c9670 0x7ff1680c9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.647 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff16ec43700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff16806ea50 con 0x7ff168158630 2023-06-23T01:08:38.647 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff16ec43700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff168157e10 con 0x7ff1680c9670 2023-06-23T01:08:38.647 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff16ec43700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1680ca000 con 0x7ff1680c7e70 2023-06-23T01:08:38.647 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff167fff700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7ff1680c7e70 0x7ff16806eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff167fff700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7ff1680c7e70 0x7ff16806eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53310/0 (socket says 172.21.15.73:53310) 2023-06-23T01:08:38.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff167fff700 1 -- 172.21.15.73:0/4223951856 learned_addr learned my addr 172.21.15.73:0/4223951856 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:38.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff16d1e0700 1 --2- 172.21.15.73:0/4223951856 >> v2:172.21.15.39:3300/0 conn(0x7ff1680c9670 0x7ff1680c9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff167fff700 1 -- 172.21.15.73:0/4223951856 >> v2:172.21.15.39:3300/0 conn(0x7ff1680c9670 msgr2=0x7ff1680c9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:38.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff167fff700 1 --2- 172.21.15.73:0/4223951856 >> v2:172.21.15.39:3300/0 conn(0x7ff1680c9670 0x7ff1680c9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff167fff700 1 -- 172.21.15.73:0/4223951856 >> v1:172.21.15.39:6789/0 conn(0x7ff168158630 legacy=0x7ff1680c7760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.648 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff167fff700 1 -- 172.21.15.73:0/4223951856 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff168158a00 con 0x7ff1680c7e70 2023-06-23T01:08:38.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff16d1e0700 1 --2- 172.21.15.73:0/4223951856 >> v2:172.21.15.39:3300/0 conn(0x7ff1680c9670 0x7ff1680c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:08:38.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.642+0000 7ff167fff700 1 --2- 172.21.15.73:0/4223951856 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1680c7e70 0x7ff16806eee0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff1680b3120 tx=0x7ff154009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e787db357298e12 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:38.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff1677fe700 1 -- 172.21.15.73:0/4223951856 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15400b860 con 0x7ff1680c7e70 2023-06-23T01:08:38.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff1677fe700 1 -- 172.21.15.73:0/4223951856 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15400b9c0 con 0x7ff1680c7e70 2023-06-23T01:08:38.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff1677fe700 1 -- 172.21.15.73:0/4223951856 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15400bd60 con 0x7ff1680c7e70 2023-06-23T01:08:38.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff16ec43700 1 -- 172.21.15.73:0/4223951856 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1680c7e70 msgr2=0x7ff16806eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff16ec43700 1 --2- 172.21.15.73:0/4223951856 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1680c7e70 0x7ff16806eee0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7ff1680b3120 tx=0x7ff154009250 comp rx=0 tx=0).stop 2023-06-23T01:08:38.649 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff16ec43700 1 -- 172.21.15.73:0/4223951856 shutdown_connections 2023-06-23T01:08:38.650 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff16ec43700 1 --2- 172.21.15.73:0/4223951856 >> v2:172.21.15.39:3300/0 conn(0x7ff1680c9670 0x7ff1680c9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.650 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff16ec43700 1 --2- 172.21.15.73:0/4223951856 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff1680c7e70 0x7ff16806eee0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.650 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff16ec43700 1 -- 172.21.15.73:0/4223951856 >> 172.21.15.73:0/4223951856 conn(0x7ff16806b250 msgr2=0x7ff16806b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:38.650 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff16ec43700 1 -- 172.21.15.73:0/4223951856 shutdown_connections 2023-06-23T01:08:38.650 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.643+0000 7ff16ec43700 1 -- 172.21.15.73:0/4223951856 wait complete. 2023-06-23T01:08:38.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.644+0000 7ff16ec43700 1 Processor -- start 2023-06-23T01:08:38.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16ec43700 1 -- start start 2023-06-23T01:08:38.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16ec43700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff1680c7e70 0x7ff16815c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.652 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16ec43700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1680c9670 0x7ff16816a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16ec43700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff168158630 0x7ff168167970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16ec43700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff168155540 con 0x7ff1680c9670 2023-06-23T01:08:38.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16ec43700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16815ce00 con 0x7ff168158630 2023-06-23T01:08:38.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16ec43700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1680b25b0 con 0x7ff1680c7e70 2023-06-23T01:08:38.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16c9df700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff1680c7e70 0x7ff16815c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16d1e0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff168158630 0x7ff168167970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.653 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff167fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1680c9670 0x7ff16816a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16c9df700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff1680c7e70 0x7ff16815c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.73:43564/0 (socket says 172.21.15.73:43564) 2023-06-23T01:08:38.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff167fff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1680c9670 0x7ff16816a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.73:52250/0 (socket says 172.21.15.73:52250) 2023-06-23T01:08:38.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16d1e0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff168158630 0x7ff168167970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53316/0 (socket says 172.21.15.73:53316) 2023-06-23T01:08:38.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.645+0000 7ff16c9df700 1 -- 172.21.15.73:0/1621583719 learned_addr learned my addr 172.21.15.73:0/1621583719 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:38.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff16c9df700 1 -- 172.21.15.73:0/1621583719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff168158630 msgr2=0x7ff168167970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff16c9df700 1 --2- 172.21.15.73:0/1621583719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff168158630 0x7ff168167970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.654 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff16c9df700 1 -- 172.21.15.73:0/1621583719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1680c9670 msgr2=0x7ff16816a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff16c9df700 1 --2- 172.21.15.73:0/1621583719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1680c9670 0x7ff16816a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff16c9df700 1 -- 172.21.15.73:0/1621583719 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1540094f0 con 0x7ff1680c7e70 2023-06-23T01:08:38.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff16c9df700 1 --2- 172.21.15.73:0/1621583719 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff1680c7e70 0x7ff16815c740 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00c4e0 tx=0x7ff15c00c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:38.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff164ff9700 1 -- 172.21.15.73:0/1621583719 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c011590 con 0x7ff1680c7e70 2023-06-23T01:08:38.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff164ff9700 1 -- 172.21.15.73:0/1621583719 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff15c011710 con 0x7ff1680c7e70 2023-06-23T01:08:38.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1680b1c80 con 0x7ff1680c7e70 2023-06-23T01:08:38.655 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff164ff9700 1 -- 172.21.15.73:0/1621583719 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15c011ad0 con 0x7ff1680c7e70 2023-06-23T01:08:38.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1680b2240 con 0x7ff1680c7e70 2023-06-23T01:08:38.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff164ff9700 1 -- 172.21.15.73:0/1621583719 <== mon.2 v2:172.21.15.73:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff15c01c6c0 con 0x7ff1680c7e70 2023-06-23T01:08:38.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.646+0000 7ff164ff9700 1 -- 172.21.15.73:0/1621583719 <== mon.2 v2:172.21.15.73:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7ff15c01cda0 con 0x7ff1680c7e70 2023-06-23T01:08:38.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.647+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff16804c1b0 con 0x7ff1680c7e70 2023-06-23T01:08:38.656 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.649+0000 7ff164ff9700 1 -- 172.21.15.73:0/1621583719 <== mon.2 v2:172.21.15.73:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7ff15c017070 con 0x7ff1680c7e70 2023-06-23T01:08:38.657 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:38.651+0000 7f5e76672180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:08:38.662 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:38.660+0000 7fab8d2b3180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:08:38.735 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.732+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cffcd6d2-b414-40b0-8967-26421b058b3e", "id": 2} v 0) v1 -- 0x7ff1680ca2c0 con 0x7ff1680c7e70 2023-06-23T01:08:38.743 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.739+0000 7ff164ff9700 1 -- 172.21.15.73:0/1621583719 <== mon.2 v2:172.21.15.73:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cffcd6d2-b414-40b0-8967-26421b058b3e", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7ff15c01c970 con 0x7ff1680c7e70 2023-06-23T01:08:38.743 INFO:teuthology.orchestra.run.smithi073.stdout:2 2023-06-23T01:08:38.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.740+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff1680c7e70 msgr2=0x7ff16815c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.740+0000 7ff16ec43700 1 --2- 172.21.15.73:0/1621583719 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff1680c7e70 0x7ff16815c740 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff15c00c4e0 tx=0x7ff15c00c7f0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.744 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.740+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 shutdown_connections 2023-06-23T01:08:38.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.740+0000 7ff16ec43700 1 --2- 172.21.15.73:0/1621583719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff168158630 0x7ff168167970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.740+0000 7ff16ec43700 1 --2- 172.21.15.73:0/1621583719 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff1680c7e70 0x7ff16815c740 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.740+0000 7ff16ec43700 1 --2- 172.21.15.73:0/1621583719 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff1680c9670 0x7ff16816a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.740+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 >> 172.21.15.73:0/1621583719 conn(0x7ff16806b250 msgr2=0x7ff168156e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:38.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.741+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 shutdown_connections 2023-06-23T01:08:38.745 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.741+0000 7ff16ec43700 1 -- 172.21.15.73:0/1621583719 wait complete. 2023-06-23T01:08:38.755 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new 5a91897e-5924-4d57-ac5e-bfd06cf4ff93 3 2023-06-23T01:08:38.793 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:38.789+0000 7f5e76672180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:08:38.801 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:38.798+0000 7fab8d2b3180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:08:38.871 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.868+0000 7f700a921700 1 Processor -- start 2023-06-23T01:08:38.871 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.868+0000 7f700a921700 1 -- start start 2023-06-23T01:08:38.872 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:38.868+0000 7f5e76672180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:08:38.873 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f700a921700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f70040b2720 0x7f70041579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f700a921700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f700415af60 0x7f700415d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f700a921700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f700415c0e0 con 0x7f70040b2af0 2023-06-23T01:08:38.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f700a921700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700415b4d0 con 0x7f70040b2720 2023-06-23T01:08:38.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f700a921700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f700415b7b0 con 0x7f700415af60 2023-06-23T01:08:38.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7008ebe700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f700415af60 0x7f700415d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7008ebe700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f700415af60 0x7f700415d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53318/0 (socket says 172.21.15.73:53318) 2023-06-23T01:08:38.874 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7008ebe700 1 -- 172.21.15.73:0/168733332 learned_addr learned my addr 172.21.15.73:0/168733332 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:38.875 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7003fff700 1 --2- 172.21.15.73:0/168733332 >> v2:172.21.15.39:3300/0 conn(0x7f70040b2720 0x7f70041579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.875 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7008ebe700 1 -- 172.21.15.73:0/168733332 >> v2:172.21.15.39:3300/0 conn(0x7f70040b2720 msgr2=0x7f70041579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:38.875 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7008ebe700 1 --2- 172.21.15.73:0/168733332 >> v2:172.21.15.39:3300/0 conn(0x7f70040b2720 0x7f70041579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.875 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7008ebe700 1 -- 172.21.15.73:0/168733332 >> v1:172.21.15.73:6789/0 conn(0x7f70040b2af0 legacy=0x7f70040cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.875 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7008ebe700 1 -- 172.21.15.73:0/168733332 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f700415ba90 con 0x7f700415af60 2023-06-23T01:08:38.875 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7008ebe700 1 --2- 172.21.15.73:0/168733332 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f700415af60 0x7f700415d390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f700415aef0 tx=0x7f6ffc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74094cdd4d074fcc server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:38.875 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7002ffd700 1 -- 172.21.15.73:0/168733332 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc00e070 con 0x7f700415af60 2023-06-23T01:08:38.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7002ffd700 1 -- 172.21.15.73:0/168733332 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ffc00ba20 con 0x7f700415af60 2023-06-23T01:08:38.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.869+0000 7f7002ffd700 1 -- 172.21.15.73:0/168733332 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ffc0092e0 con 0x7f700415af60 2023-06-23T01:08:38.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.870+0000 7f700a921700 1 -- 172.21.15.73:0/168733332 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f700415af60 msgr2=0x7f700415d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.870+0000 7f700a921700 1 --2- 172.21.15.73:0/168733332 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f700415af60 0x7f700415d390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f700415aef0 tx=0x7f6ffc009840 comp rx=0 tx=0).stop 2023-06-23T01:08:38.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.870+0000 7f700a921700 1 -- 172.21.15.73:0/168733332 shutdown_connections 2023-06-23T01:08:38.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.870+0000 7f700a921700 1 --2- 172.21.15.73:0/168733332 >> v2:172.21.15.39:3300/0 conn(0x7f70040b2720 0x7f70041579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.870+0000 7f700a921700 1 --2- 172.21.15.73:0/168733332 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f700415af60 0x7f700415d390 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.876 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.870+0000 7f700a921700 1 -- 172.21.15.73:0/168733332 >> 172.21.15.73:0/168733332 conn(0x7f7004067450 msgr2=0x7f7004067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:38.877 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.870+0000 7f700a921700 1 -- 172.21.15.73:0/168733332 shutdown_connections 2023-06-23T01:08:38.877 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.870+0000 7f700a921700 1 -- 172.21.15.73:0/168733332 wait complete. 2023-06-23T01:08:38.878 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.871+0000 7f700a921700 1 Processor -- start 2023-06-23T01:08:38.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.871+0000 7f700a921700 1 -- start start 2023-06-23T01:08:38.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f700a921700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f70040b2720 0x7f7004145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f700a921700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f70040b2af0 0x7f700415f200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f700a921700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700415af60 0x7f700415f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:38.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f700a921700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040b3a10 con 0x7f700415af60 2023-06-23T01:08:38.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f700a921700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70040c6e40 con 0x7f70040b2720 2023-06-23T01:08:38.879 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f700a921700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70040c6b60 con 0x7f70040b2af0 2023-06-23T01:08:38.880 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f7003fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f70040b2720 0x7f7004145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.880 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f70037fe700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f70040b2af0 0x7f700415f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.880 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f7003fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f70040b2720 0x7f7004145fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53334/0 (socket says 172.21.15.73:53334) 2023-06-23T01:08:38.880 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f7003fff700 1 -- 172.21.15.73:0/4011568867 learned_addr learned my addr 172.21.15.73:0/4011568867 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:38.880 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f7008ebe700 1 --2- 172.21.15.73:0/4011568867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700415af60 0x7f700415f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:38.880 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f70037fe700 1 -- 172.21.15.73:0/4011568867 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f70040b2720 msgr2=0x7f7004145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.880 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f70037fe700 1 --2- 172.21.15.73:0/4011568867 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f70040b2720 0x7f7004145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f70037fe700 1 -- 172.21.15.73:0/4011568867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700415af60 msgr2=0x7f700415f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f70037fe700 1 --2- 172.21.15.73:0/4011568867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700415af60 0x7f700415f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.872+0000 7f70037fe700 1 -- 172.21.15.73:0/4011568867 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ffc009ae0 con 0x7f70040b2af0 2023-06-23T01:08:38.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.873+0000 7f70037fe700 1 --2- 172.21.15.73:0/4011568867 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f70040b2af0 0x7f700415f200 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400c590 tx=0x7f6ff4007ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:38.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.873+0000 7f6febfff700 1 -- 172.21.15.73:0/4011568867 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4011590 con 0x7f70040b2af0 2023-06-23T01:08:38.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.873+0000 7f6febfff700 1 -- 172.21.15.73:0/4011568867 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ff4011710 con 0x7f70040b2af0 2023-06-23T01:08:38.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.873+0000 7f6febfff700 1 -- 172.21.15.73:0/4011568867 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ff4011ad0 con 0x7f70040b2af0 2023-06-23T01:08:38.881 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.873+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70040ce8f0 con 0x7f70040b2af0 2023-06-23T01:08:38.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.873+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70040ceeb0 con 0x7f70040b2af0 2023-06-23T01:08:38.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.873+0000 7f6febfff700 1 -- 172.21.15.73:0/4011568867 <== mon.2 v2:172.21.15.73:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6ff401c6c0 con 0x7f70040b2af0 2023-06-23T01:08:38.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.873+0000 7f6febfff700 1 -- 172.21.15.73:0/4011568867 <== mon.2 v2:172.21.15.73:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f6ff40128c0 con 0x7f70040b2af0 2023-06-23T01:08:38.882 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.874+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f700404c1b0 con 0x7f70040b2af0 2023-06-23T01:08:38.882 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:38.877+0000 7fab8d2b3180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:08:38.883 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.876+0000 7f6febfff700 1 -- 172.21.15.73:0/4011568867 <== mon.2 v2:172.21.15.73:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f6ff4017070 con 0x7f70040b2af0 2023-06-23T01:08:38.961 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.957+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "osd new", "uuid": "5a91897e-5924-4d57-ac5e-bfd06cf4ff93", "id": 3} v 0) v1 -- 0x7f70040cf0a0 con 0x7f70040b2af0 2023-06-23T01:08:38.973 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.969+0000 7f6febfff700 1 -- 172.21.15.73:0/4011568867 <== mon.2 v2:172.21.15.73:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5a91897e-5924-4d57-ac5e-bfd06cf4ff93", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6ff401cba0 con 0x7f70040b2af0 2023-06-23T01:08:38.973 INFO:teuthology.orchestra.run.smithi073.stdout:3 2023-06-23T01:08:38.974 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f70040b2af0 msgr2=0x7f700415f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:38.974 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 --2- 172.21.15.73:0/4011568867 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f70040b2af0 0x7f700415f200 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6ff400c590 tx=0x7f6ff4007ec0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.974 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 shutdown_connections 2023-06-23T01:08:38.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 --2- 172.21.15.73:0/4011568867 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f700415af60 0x7f700415f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 --2- 172.21.15.73:0/4011568867 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f70040b2720 0x7f7004145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 --2- 172.21.15.73:0/4011568867 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f70040b2af0 0x7f700415f200 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:38.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 >> 172.21.15.73:0/4011568867 conn(0x7f7004067450 msgr2=0x7f70040ccc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:38.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 shutdown_connections 2023-06-23T01:08:38.975 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:38.970+0000 7f700a921700 1 -- 172.21.15.73:0/4011568867 wait complete. 2023-06-23T01:08:38.985 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new 67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5 4 2023-06-23T01:08:39.062 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:39.059+0000 7f5e76672180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:08:39.072 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:39.069+0000 7fab8d2b3180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:08:39.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.137+0000 7fb495665700 1 Processor -- start 2023-06-23T01:08:39.140 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.137+0000 7fb495665700 1 -- start start 2023-06-23T01:08:39.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb495665700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fb4900b2af0 0x7fb4900cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb495665700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb49015af60 0x7fb49015d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb495665700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49015c0e0 con 0x7fb4900b2720 2023-06-23T01:08:39.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb495665700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb49015b4d0 con 0x7fb4900b2af0 2023-06-23T01:08:39.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb495665700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49015b7b0 con 0x7fb49015af60 2023-06-23T01:08:39.142 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48e7fc700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fb4900b2af0 0x7fb4900cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48f7fe700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb49015af60 0x7fb49015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48e7fc700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fb4900b2af0 0x7fb4900cd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53342/0 (socket says 172.21.15.73:53342) 2023-06-23T01:08:39.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48f7fe700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb49015af60 0x7fb49015d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.73:43588/0 (socket says 172.21.15.73:43588) 2023-06-23T01:08:39.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48e7fc700 1 -- 172.21.15.73:0/2236911260 learned_addr learned my addr 172.21.15.73:0/2236911260 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:39.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48e7fc700 1 -- 172.21.15.73:0/2236911260 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb49015af60 msgr2=0x7fb49015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:39.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48e7fc700 1 --2- 172.21.15.73:0/2236911260 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb49015af60 0x7fb49015d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.143 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48e7fc700 1 -- 172.21.15.73:0/2236911260 >> v1:172.21.15.73:6789/0 conn(0x7fb4900b2720 legacy=0x7fb4901579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48e7fc700 1 -- 172.21.15.73:0/2236911260 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49015ba90 con 0x7fb4900b2af0 2023-06-23T01:08:39.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48f7fe700 1 --2- 172.21.15.73:0/2236911260 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb49015af60 0x7fb49015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:39.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48e7fc700 1 --2- 172.21.15.73:0/2236911260 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4900b2af0 0x7fb4900cd6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb4840099b0 tx=0x7fb484009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68e38991412caabf server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:39.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48dffb700 1 -- 172.21.15.73:0/2236911260 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb484009ea0 con 0x7fb4900b2af0 2023-06-23T01:08:39.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48dffb700 1 -- 172.21.15.73:0/2236911260 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48400b860 con 0x7fb4900b2af0 2023-06-23T01:08:39.144 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.138+0000 7fb48dffb700 1 -- 172.21.15.73:0/2236911260 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48400bc00 con 0x7fb4900b2af0 2023-06-23T01:08:39.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.139+0000 7fb495665700 1 -- 172.21.15.73:0/2236911260 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4900b2af0 msgr2=0x7fb4900cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.139+0000 7fb495665700 1 --2- 172.21.15.73:0/2236911260 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4900b2af0 0x7fb4900cd6c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb4840099b0 tx=0x7fb484009250 comp rx=0 tx=0).stop 2023-06-23T01:08:39.146 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.139+0000 7fb495665700 1 -- 172.21.15.73:0/2236911260 shutdown_connections 2023-06-23T01:08:39.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.139+0000 7fb495665700 1 --2- 172.21.15.73:0/2236911260 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb49015af60 0x7fb49015d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.139+0000 7fb495665700 1 --2- 172.21.15.73:0/2236911260 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4900b2af0 0x7fb4900cd6c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.139+0000 7fb495665700 1 -- 172.21.15.73:0/2236911260 >> 172.21.15.73:0/2236911260 conn(0x7fb490067450 msgr2=0x7fb490067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:39.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.139+0000 7fb495665700 1 -- 172.21.15.73:0/2236911260 shutdown_connections 2023-06-23T01:08:39.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.139+0000 7fb495665700 1 -- 172.21.15.73:0/2236911260 wait complete. 2023-06-23T01:08:39.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.140+0000 7fb495665700 1 Processor -- start 2023-06-23T01:08:39.147 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.140+0000 7fb495665700 1 -- start start 2023-06-23T01:08:39.148 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb495665700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4900b2720 0x7fb490149190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.148 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb495665700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb4900b2af0 0x7fb4900b31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.148 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb495665700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb49015af60 0x7fb4900b1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.148 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb495665700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4901496d0 con 0x7fb4900b2720 2023-06-23T01:08:39.148 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb495665700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900c7bf0 con 0x7fb49015af60 2023-06-23T01:08:39.148 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb495665700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4900c7d80 con 0x7fb4900b2af0 2023-06-23T01:08:39.148 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48e7fc700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb4900b2af0 0x7fb4900b31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb49015af60 0x7fb4900b1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48e7fc700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb4900b2af0 0x7fb4900b31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.73:43598/0 (socket says 172.21.15.73:43598) 2023-06-23T01:08:39.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb49015af60 0x7fb4900b1140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53356/0 (socket says 172.21.15.73:53356) 2023-06-23T01:08:39.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48e7fc700 1 -- 172.21.15.73:0/3866256137 learned_addr learned my addr 172.21.15.73:0/3866256137 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:39.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48effd700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4900b2720 0x7fb490149190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48f7fe700 1 -- 172.21.15.73:0/3866256137 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb4900b2af0 msgr2=0x7fb4900b31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48f7fe700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb4900b2af0 0x7fb4900b31e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.149 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48f7fe700 1 -- 172.21.15.73:0/3866256137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4900b2720 msgr2=0x7fb490149190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48f7fe700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4900b2720 0x7fb490149190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48f7fe700 1 -- 172.21.15.73:0/3866256137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb488009a20 con 0x7fb49015af60 2023-06-23T01:08:39.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48e7fc700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb4900b2af0 0x7fb4900b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:39.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.141+0000 7fb48effd700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4900b2720 0x7fb490149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:08:39.150 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.142+0000 7fb48f7fe700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb49015af60 0x7fb4900b1140 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb488009e70 tx=0x7fb48800b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:39.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.142+0000 7fb476ffd700 1 -- 172.21.15.73:0/3866256137 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb48800cb50 con 0x7fb49015af60 2023-06-23T01:08:39.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.142+0000 7fb476ffd700 1 -- 172.21.15.73:0/3866256137 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb48800f040 con 0x7fb49015af60 2023-06-23T01:08:39.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.142+0000 7fb476ffd700 1 -- 172.21.15.73:0/3866256137 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb488014480 con 0x7fb49015af60 2023-06-23T01:08:39.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.142+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4840094f0 con 0x7fb49015af60 2023-06-23T01:08:39.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.142+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4900b41f0 con 0x7fb49015af60 2023-06-23T01:08:39.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.142+0000 7fb476ffd700 1 -- 172.21.15.73:0/3866256137 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb488003020 con 0x7fb49015af60 2023-06-23T01:08:39.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.142+0000 7fb476ffd700 1 -- 172.21.15.73:0/3866256137 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7fb4880138f0 con 0x7fb49015af60 2023-06-23T01:08:39.151 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.143+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb49004c1b0 con 0x7fb49015af60 2023-06-23T01:08:39.152 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.146+0000 7fb476ffd700 1 -- 172.21.15.73:0/3866256137 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fb488018030 con 0x7fb49015af60 2023-06-23T01:08:39.231 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.227+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd new", "uuid": "67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5", "id": 4} v 0) v1 -- 0x7fb4900c7ec0 con 0x7fb49015af60 2023-06-23T01:08:39.242 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:39.239+0000 7f5e76672180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:08:39.243 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.240+0000 7fb476ffd700 1 -- 172.21.15.73:0/3866256137 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb488014990 con 0x7fb49015af60 2023-06-23T01:08:39.243 INFO:teuthology.orchestra.run.smithi073.stdout:4 2023-06-23T01:08:39.244 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb49015af60 msgr2=0x7fb4900b1140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.245 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb49015af60 0x7fb4900b1140 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb488009e70 tx=0x7fb48800b040 comp rx=0 tx=0).stop 2023-06-23T01:08:39.245 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 shutdown_connections 2023-06-23T01:08:39.245 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb49015af60 0x7fb4900b1140 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.245 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb4900b2720 0x7fb490149190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.245 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 --2- 172.21.15.73:0/3866256137 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb4900b2af0 0x7fb4900b31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.245 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 >> 172.21.15.73:0/3866256137 conn(0x7fb490067450 msgr2=0x7fb490154c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:39.245 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 shutdown_connections 2023-06-23T01:08:39.246 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.241+0000 7fb495665700 1 -- 172.21.15.73:0/3866256137 wait complete. 2023-06-23T01:08:39.254 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:39.251+0000 7fab8d2b3180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:08:39.255 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new 390058ef-ef47-475e-af72-8e2f51ee184c 5 2023-06-23T01:08:39.324 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:39.321+0000 7f5e76672180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:08:39.335 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:39.332+0000 7fab8d2b3180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:08:39.414 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.411+0000 7f7b8e14d700 1 Processor -- start 2023-06-23T01:08:39.415 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.411+0000 7f7b8e14d700 1 -- start start 2023-06-23T01:08:39.416 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8e14d700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7b88156ca0 0x7f7b8815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.417 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8e14d700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f7b88067b60 0x7f7b8815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.417 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8e14d700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b880ce670 con 0x7f7b88157070 2023-06-23T01:08:39.417 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8e14d700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8815de90 con 0x7f7b88156ca0 2023-06-23T01:08:39.417 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8e14d700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b8815d3e0 con 0x7f7b88067b60 2023-06-23T01:08:39.417 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8d94c700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f7b88067b60 0x7f7b8815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.417 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8d14b700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7b88156ca0 0x7f7b8815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.418 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8d94c700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f7b88067b60 0x7f7b8815cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53364/0 (socket says 172.21.15.73:53364) 2023-06-23T01:08:39.418 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8d94c700 1 -- 172.21.15.73:0/2312767612 learned_addr learned my addr 172.21.15.73:0/2312767612 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:39.418 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.412+0000 7f7b8c94a700 1 --1- 172.21.15.73:0/2312767612 >> v1:172.21.15.39:6789/0 conn(0x7f7b88157070 0x7f7b88067450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.73:53388/0 (socket says 172.21.15.73:53388) 2023-06-23T01:08:39.418 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.413+0000 7f7b8d94c700 1 -- 172.21.15.73:0/2312767612 >> v2:172.21.15.39:3300/0 conn(0x7f7b88156ca0 msgr2=0x7f7b8815a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:39.418 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.413+0000 7f7b8d94c700 1 --2- 172.21.15.73:0/2312767612 >> v2:172.21.15.39:3300/0 conn(0x7f7b88156ca0 0x7f7b8815a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.418 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.413+0000 7f7b8d94c700 1 -- 172.21.15.73:0/2312767612 >> v1:172.21.15.39:6789/0 conn(0x7f7b88157070 legacy=0x7f7b88067450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.419 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.413+0000 7f7b8d94c700 1 -- 172.21.15.73:0/2312767612 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8815d5f0 con 0x7f7b88067b60 2023-06-23T01:08:39.419 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.413+0000 7f7b8d94c700 1 --2- 172.21.15.73:0/2312767612 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88067b60 0x7f7b8815cea0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7b881598a0 tx=0x7f7b80009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38c962347777fe39 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:39.419 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.413+0000 7f7b7ffff700 1 -- 172.21.15.73:0/2312767612 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b8000e070 con 0x7f7b88067b60 2023-06-23T01:08:39.419 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.413+0000 7f7b7ffff700 1 -- 172.21.15.73:0/2312767612 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b8000ba20 con 0x7f7b88067b60 2023-06-23T01:08:39.419 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.413+0000 7f7b7ffff700 1 -- 172.21.15.73:0/2312767612 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b800092e0 con 0x7f7b88067b60 2023-06-23T01:08:39.422 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.414+0000 7f7b8e14d700 1 -- 172.21.15.73:0/2312767612 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88067b60 msgr2=0x7f7b8815cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.422 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.414+0000 7f7b8e14d700 1 --2- 172.21.15.73:0/2312767612 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88067b60 0x7f7b8815cea0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7b881598a0 tx=0x7f7b80009840 comp rx=0 tx=0).stop 2023-06-23T01:08:39.422 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.414+0000 7f7b8e14d700 1 -- 172.21.15.73:0/2312767612 shutdown_connections 2023-06-23T01:08:39.422 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.414+0000 7f7b8e14d700 1 --2- 172.21.15.73:0/2312767612 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88067b60 0x7f7b8815cea0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.422 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.414+0000 7f7b8e14d700 1 --2- 172.21.15.73:0/2312767612 >> v2:172.21.15.39:3300/0 conn(0x7f7b88156ca0 0x7f7b8815a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.414+0000 7f7b8e14d700 1 -- 172.21.15.73:0/2312767612 >> 172.21.15.73:0/2312767612 conn(0x7f7b880d0240 msgr2=0x7f7b880d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:39.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.414+0000 7f7b8e14d700 1 -- 172.21.15.73:0/2312767612 shutdown_connections 2023-06-23T01:08:39.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.414+0000 7f7b8e14d700 1 -- 172.21.15.73:0/2312767612 wait complete. 2023-06-23T01:08:39.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.415+0000 7f7b8e14d700 1 Processor -- start 2023-06-23T01:08:39.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.415+0000 7f7b8e14d700 1 -- start start 2023-06-23T01:08:39.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8e14d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b88067b60 0x7f7b88154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.423 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8e14d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88156ca0 0x7f7b880b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8e14d700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7b88157070 0x7f7b880b4180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8e14d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b880c5990 con 0x7f7b88067b60 2023-06-23T01:08:39.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8e14d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b880c6780 con 0x7f7b88156ca0 2023-06-23T01:08:39.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8e14d700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b880c5b70 con 0x7f7b88157070 2023-06-23T01:08:39.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88156ca0 0x7f7b880b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.424 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88156ca0 0x7f7b880b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53370/0 (socket says 172.21.15.73:53370) 2023-06-23T01:08:39.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 -- 172.21.15.73:0/3302427082 learned_addr learned my addr 172.21.15.73:0/3302427082 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:39.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8d14b700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b88067b60 0x7f7b88154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8d94c700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7b88157070 0x7f7b880b4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 -- 172.21.15.73:0/3302427082 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7b88157070 msgr2=0x7f7b880b4180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7b88157070 0x7f7b880b4180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 -- 172.21.15.73:0/3302427082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b88067b60 msgr2=0x7f7b88154020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.425 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b88067b60 0x7f7b88154020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 -- 172.21.15.73:0/3302427082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b80009ae0 con 0x7f7b88156ca0 2023-06-23T01:08:39.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8c94a700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88156ca0 0x7f7b880b3c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7b84001eb0 tx=0x7f7b84007f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:39.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b7d7fa700 1 -- 172.21.15.73:0/3302427082 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b84005a70 con 0x7f7b88156ca0 2023-06-23T01:08:39.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b7d7fa700 1 -- 172.21.15.73:0/3302427082 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b84005bd0 con 0x7f7b88156ca0 2023-06-23T01:08:39.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b880c5e50 con 0x7f7b88156ca0 2023-06-23T01:08:39.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b7d7fa700 1 -- 172.21.15.73:0/3302427082 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b840070d0 con 0x7f7b88156ca0 2023-06-23T01:08:39.426 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.416+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b880c6410 con 0x7f7b88156ca0 2023-06-23T01:08:39.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.417+0000 7f7b7d7fa700 1 -- 172.21.15.73:0/3302427082 <== mon.1 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7b8401c440 con 0x7f7b88156ca0 2023-06-23T01:08:39.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.417+0000 7f7b7d7fa700 1 -- 172.21.15.73:0/3302427082 <== mon.1 v2:172.21.15.73:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f7b84007700 con 0x7f7b88156ca0 2023-06-23T01:08:39.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.417+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8804c1b0 con 0x7f7b88156ca0 2023-06-23T01:08:39.427 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.419+0000 7f7b7d7fa700 1 -- 172.21.15.73:0/3302427082 <== mon.1 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f7b84017070 con 0x7f7b88156ca0 2023-06-23T01:08:39.466 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:39.462+0000 7f5e76672180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:08:39.476 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:39.473+0000 7fab8d2b3180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:08:39.505 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.502+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd new", "uuid": "390058ef-ef47-475e-af72-8e2f51ee184c", "id": 5} v 0) v1 -- 0x7f7b880c68c0 con 0x7f7b88156ca0 2023-06-23T01:08:39.511 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.508+0000 7f7b7d7fa700 1 -- 172.21.15.73:0/3302427082 <== mon.1 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "390058ef-ef47-475e-af72-8e2f51ee184c", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7b8401c880 con 0x7f7b88156ca0 2023-06-23T01:08:39.511 INFO:teuthology.orchestra.run.smithi073.stdout:5 2023-06-23T01:08:39.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88156ca0 msgr2=0x7f7b880b3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88156ca0 0x7f7b880b3c40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7b84001eb0 tx=0x7f7b84007f40 comp rx=0 tx=0).stop 2023-06-23T01:08:39.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 shutdown_connections 2023-06-23T01:08:39.512 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7b88157070 0x7f7b880b4180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7b88067b60 0x7f7b88154020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 --2- 172.21.15.73:0/3302427082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b88156ca0 0x7f7b880b3c40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 >> 172.21.15.73:0/3302427082 conn(0x7f7b880d0240 msgr2=0x7f7b88155ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:39.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 shutdown_connections 2023-06-23T01:08:39.513 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.509+0000 7f7b8e14d700 1 -- 172.21.15.73:0/3302427082 wait complete. 2023-06-23T01:08:39.524 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new c34ad563-5018-4cf4-830f-2ac56a3a5e5f 6 2023-06-23T01:08:39.566 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:39.562+0000 7f5e76672180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:08:39.576 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:39.573+0000 7fab8d2b3180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:08:39.680 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.677+0000 7f3c23458700 1 Processor -- start 2023-06-23T01:08:39.680 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.677+0000 7f3c23458700 1 -- start start 2023-06-23T01:08:39.681 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c23458700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f3c1c156d20 0x7f3c1c1570f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c23458700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c15a7a0 0x7f3c1c06b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c23458700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cbdf0 0x7f3c1c0cc290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c23458700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c06d480 con 0x7f3c1c0cbdf0 2023-06-23T01:08:39.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c23458700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c0cc7d0 con 0x7f3c1c156d20 2023-06-23T01:08:39.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c23458700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c06d910 con 0x7f3c1c15a7a0 2023-06-23T01:08:39.682 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c211f4700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f3c1c156d20 0x7f3c1c1570f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c209f3700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c15a7a0 0x7f3c1c06b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c211f4700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f3c1c156d20 0x7f3c1c1570f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53372/0 (socket says 172.21.15.73:53372) 2023-06-23T01:08:39.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c211f4700 1 -- 172.21.15.73:0/1566228025 learned_addr learned my addr 172.21.15.73:0/1566228025 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:39.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c209f3700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c15a7a0 0x7f3c1c06b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.73:43608/0 (socket says 172.21.15.73:43608) 2023-06-23T01:08:39.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c219f5700 1 --2- 172.21.15.73:0/1566228025 >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cbdf0 0x7f3c1c0cc290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c209f3700 1 -- 172.21.15.73:0/1566228025 >> v2:172.21.15.73:3300/0 conn(0x7f3c1c156d20 msgr2=0x7f3c1c1570f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:39.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c209f3700 1 --2- 172.21.15.73:0/1566228025 >> v2:172.21.15.73:3300/0 conn(0x7f3c1c156d20 0x7f3c1c1570f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.683 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c209f3700 1 -- 172.21.15.73:0/1566228025 >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cbdf0 msgr2=0x7f3c1c0cc290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:39.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c209f3700 1 --2- 172.21.15.73:0/1566228025 >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cbdf0 0x7f3c1c0cc290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c209f3700 1 -- 172.21.15.73:0/1566228025 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c06bef0 con 0x7f3c1c15a7a0 2023-06-23T01:08:39.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c211f4700 1 --2- 172.21.15.73:0/1566228025 >> v2:172.21.15.73:3300/0 conn(0x7f3c1c156d20 0x7f3c1c1570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:39.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c209f3700 1 --2- 172.21.15.73:0/1566228025 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c15a7a0 0x7f3c1c06b580 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c06af40 tx=0x7f3c0c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca3a29b138290016 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:39.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c13fff700 1 -- 172.21.15.73:0/1566228025 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c011630 con 0x7f3c1c15a7a0 2023-06-23T01:08:39.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c13fff700 1 -- 172.21.15.73:0/1566228025 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c0117b0 con 0x7f3c1c15a7a0 2023-06-23T01:08:39.684 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.678+0000 7f3c13fff700 1 -- 172.21.15.73:0/1566228025 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c011b70 con 0x7f3c1c15a7a0 2023-06-23T01:08:39.686 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 -- 172.21.15.73:0/1566228025 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c15a7a0 msgr2=0x7f3c1c06b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 --2- 172.21.15.73:0/1566228025 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c15a7a0 0x7f3c1c06b580 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3c1c06af40 tx=0x7f3c0c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 -- 172.21.15.73:0/1566228025 shutdown_connections 2023-06-23T01:08:39.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 --2- 172.21.15.73:0/1566228025 >> v2:172.21.15.39:3300/0 conn(0x7f3c1c0cbdf0 0x7f3c1c0cc290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 --2- 172.21.15.73:0/1566228025 >> v2:172.21.15.73:3300/0 conn(0x7f3c1c156d20 0x7f3c1c1570f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 --2- 172.21.15.73:0/1566228025 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c15a7a0 0x7f3c1c06b580 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.687 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 -- 172.21.15.73:0/1566228025 >> 172.21.15.73:0/1566228025 conn(0x7f3c1c0ce220 msgr2=0x7f3c1c0ce620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:39.688 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 -- 172.21.15.73:0/1566228025 shutdown_connections 2023-06-23T01:08:39.688 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.679+0000 7f3c23458700 1 -- 172.21.15.73:0/1566228025 wait complete. 2023-06-23T01:08:39.688 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.680+0000 7f3c23458700 1 Processor -- start 2023-06-23T01:08:39.688 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.680+0000 7f3c23458700 1 -- start start 2023-06-23T01:08:39.688 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c23458700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0cbdf0 0x7f3c1c141e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.688 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c23458700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c156d20 0x7f3c1c142350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.688 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c23458700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c1c15a7a0 0x7f3c1c0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c23458700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0c5600 con 0x7f3c1c0cbdf0 2023-06-23T01:08:39.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c23458700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c143600 con 0x7f3c1c15a7a0 2023-06-23T01:08:39.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c23458700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c1429f0 con 0x7f3c1c156d20 2023-06-23T01:08:39.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c209f3700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c156d20 0x7f3c1c142350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c209f3700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c156d20 0x7f3c1c142350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.73:43610/0 (socket says 172.21.15.73:43610) 2023-06-23T01:08:39.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c219f5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c1c15a7a0 0x7f3c1c0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c209f3700 1 -- 172.21.15.73:0/4090875476 learned_addr learned my addr 172.21.15.73:0/4090875476 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:39.689 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.681+0000 7f3c211f4700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0cbdf0 0x7f3c1c141e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c209f3700 1 -- 172.21.15.73:0/4090875476 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c1c15a7a0 msgr2=0x7f3c1c0b1f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c209f3700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c1c15a7a0 0x7f3c1c0b1f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c209f3700 1 -- 172.21.15.73:0/4090875476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0cbdf0 msgr2=0x7f3c1c141e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c209f3700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0cbdf0 0x7f3c1c141e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c209f3700 1 -- 172.21.15.73:0/4090875476 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c180094f0 con 0x7f3c1c156d20 2023-06-23T01:08:39.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c219f5700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c1c15a7a0 0x7f3c1c0b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:39.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c209f3700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c156d20 0x7f3c1c142350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c013000 tx=0x7f3c0c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:39.690 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c117fa700 1 -- 172.21.15.73:0/4090875476 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c01f070 con 0x7f3c1c156d20 2023-06-23T01:08:39.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c117fa700 1 -- 172.21.15.73:0/4090875476 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c000bc0 con 0x7f3c1c156d20 2023-06-23T01:08:39.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0c009420 con 0x7f3c1c156d20 2023-06-23T01:08:39.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c117fa700 1 -- 172.21.15.73:0/4090875476 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0c011b50 con 0x7f3c1c156d20 2023-06-23T01:08:39.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c142fb0 con 0x7f3c1c156d20 2023-06-23T01:08:39.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c117fa700 1 -- 172.21.15.73:0/4090875476 <== mon.2 v2:172.21.15.73:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3c0c023070 con 0x7f3c1c156d20 2023-06-23T01:08:39.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.682+0000 7f3c117fa700 1 -- 172.21.15.73:0/4090875476 <== mon.2 v2:172.21.15.73:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f3c0c02e9b0 con 0x7f3c1c156d20 2023-06-23T01:08:39.691 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.683+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c143770 con 0x7f3c1c156d20 2023-06-23T01:08:39.692 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.685+0000 7f3c117fa700 1 -- 172.21.15.73:0/4090875476 <== mon.2 v2:172.21.15.73:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f3c0c02a070 con 0x7f3c1c156d20 2023-06-23T01:08:39.770 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.767+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "osd new", "uuid": "c34ad563-5018-4cf4-830f-2ac56a3a5e5f", "id": 6} v 0) v1 -- 0x7f3c1c04c1b0 con 0x7f3c1c156d20 2023-06-23T01:08:39.780 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.777+0000 7f3c117fa700 1 -- 172.21.15.73:0/4090875476 <== mon.2 v2:172.21.15.73:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c34ad563-5018-4cf4-830f-2ac56a3a5e5f", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3c0c015020 con 0x7f3c1c156d20 2023-06-23T01:08:39.781 INFO:teuthology.orchestra.run.smithi073.stdout:6 2023-06-23T01:08:39.781 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c156d20 msgr2=0x7f3c1c142350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c156d20 0x7f3c1c142350 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3c0c013000 tx=0x7f3c0c017000 comp rx=0 tx=0).stop 2023-06-23T01:08:39.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 shutdown_connections 2023-06-23T01:08:39.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c1c15a7a0 0x7f3c1c0b1f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f3c1c0cbdf0 0x7f3c1c141e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 --2- 172.21.15.73:0/4090875476 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3c1c156d20 0x7f3c1c142350 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 >> 172.21.15.73:0/4090875476 conn(0x7f3c1c0ce220 msgr2=0x7f3c1c0c4730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:39.782 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 shutdown_connections 2023-06-23T01:08:39.783 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.778+0000 7f3c23458700 1 -- 172.21.15.73:0/4090875476 wait complete. 2023-06-23T01:08:39.793 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster ceph osd new e2023d49-2e44-412c-8cd1-c67447567c32 7 2023-06-23T01:08:39.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.908+0000 7f7f6428b700 1 Processor -- start 2023-06-23T01:08:39.911 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.908+0000 7f7f6428b700 1 -- start start 2023-06-23T01:08:39.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.909+0000 7f7f6428b700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f7f5c0b2af0 0x7f7f5c0cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.913 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.909+0000 7f7f6428b700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c15d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.909+0000 7f7f6428b700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c15c0e0 con 0x7f7f5c0b2720 2023-06-23T01:08:39.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.909+0000 7f7f6428b700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c15b4d0 con 0x7f7f5c0b2af0 2023-06-23T01:08:39.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.909+0000 7f7f6428b700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c15b7b0 con 0x7f7f5c15af60 2023-06-23T01:08:39.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f62828700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f62828700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.73:43622/0 (socket says 172.21.15.73:43622) 2023-06-23T01:08:39.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f62828700 1 -- 172.21.15.73:0/3642011570 learned_addr learned my addr 172.21.15.73:0/3642011570 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:39.914 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f61826700 1 --2- 172.21.15.73:0/3642011570 >> v2:172.21.15.39:3300/0 conn(0x7f7f5c0b2af0 0x7f7f5c0cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f62828700 1 -- 172.21.15.73:0/3642011570 >> v2:172.21.15.39:3300/0 conn(0x7f7f5c0b2af0 msgr2=0x7f7f5c0cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f62828700 1 --2- 172.21.15.73:0/3642011570 >> v2:172.21.15.39:3300/0 conn(0x7f7f5c0b2af0 0x7f7f5c0cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f62828700 1 -- 172.21.15.73:0/3642011570 >> v1:172.21.15.39:6789/0 conn(0x7f7f5c0b2720 legacy=0x7f7f5c1579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f62828700 1 -- 172.21.15.73:0/3642011570 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c15ba90 con 0x7f7f5c15af60 2023-06-23T01:08:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f62828700 1 --2- 172.21.15.73:0/3642011570 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c15d390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009f20 tx=0x7f7f54009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eecb1ebcf78fca8 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f61025700 1 -- 172.21.15.73:0/3642011570 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f5400e070 con 0x7f7f5c15af60 2023-06-23T01:08:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f61025700 1 -- 172.21.15.73:0/3642011570 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f5400b9d0 con 0x7f7f5c15af60 2023-06-23T01:08:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.910+0000 7f7f61025700 1 -- 172.21.15.73:0/3642011570 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f540092e0 con 0x7f7f5c15af60 2023-06-23T01:08:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.911+0000 7f7f6428b700 1 -- 172.21.15.73:0/3642011570 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 msgr2=0x7f7f5c15d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.911+0000 7f7f6428b700 1 --2- 172.21.15.73:0/3642011570 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c15d390 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f7f54009f20 tx=0x7f7f54009840 comp rx=0 tx=0).stop 2023-06-23T01:08:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.911+0000 7f7f6428b700 1 -- 172.21.15.73:0/3642011570 shutdown_connections 2023-06-23T01:08:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.911+0000 7f7f6428b700 1 --2- 172.21.15.73:0/3642011570 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c15d390 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.911+0000 7f7f6428b700 1 --2- 172.21.15.73:0/3642011570 >> v2:172.21.15.39:3300/0 conn(0x7f7f5c0b2af0 0x7f7f5c0cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.911+0000 7f7f6428b700 1 -- 172.21.15.73:0/3642011570 >> 172.21.15.73:0/3642011570 conn(0x7f7f5c067450 msgr2=0x7f7f5c067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.911+0000 7f7f6428b700 1 -- 172.21.15.73:0/3642011570 shutdown_connections 2023-06-23T01:08:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.911+0000 7f7f6428b700 1 -- 172.21.15.73:0/3642011570 wait complete. 2023-06-23T01:08:39.918 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.913+0000 7f7f6428b700 1 Processor -- start 2023-06-23T01:08:39.918 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.913+0000 7f7f6428b700 1 -- start start 2023-06-23T01:08:39.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.914+0000 7f7f6428b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f5c0b2720 0x7f7f5c1441c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.919 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.914+0000 7f7f6428b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0b2af0 0x7f7f5c0b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.914+0000 7f7f6428b700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c1569e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:39.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.914+0000 7f7f6428b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c157480 con 0x7f7f5c0b2720 2023-06-23T01:08:39.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.914+0000 7f7f6428b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0cfad0 con 0x7f7f5c0b2af0 2023-06-23T01:08:39.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.914+0000 7f7f6428b700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0cf1a0 con 0x7f7f5c15af60 2023-06-23T01:08:39.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.914+0000 7f7f62828700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c1569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f61826700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0b2af0 0x7f7f5c0b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.920 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62828700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c1569e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3301/0 says I am v2:172.21.15.73:43636/0 (socket says 172.21.15.73:43636) 2023-06-23T01:08:39.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62828700 1 -- 172.21.15.73:0/974547367 learned_addr learned my addr 172.21.15.73:0/974547367 (peer_addr_for_me v2:172.21.15.73:0/0) 2023-06-23T01:08:39.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62027700 1 --2- 172.21.15.73:0/974547367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f5c0b2720 0x7f7f5c1441c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:39.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62828700 1 -- 172.21.15.73:0/974547367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0b2af0 msgr2=0x7f7f5c0b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62828700 1 --2- 172.21.15.73:0/974547367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0b2af0 0x7f7f5c0b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62828700 1 -- 172.21.15.73:0/974547367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f5c0b2720 msgr2=0x7f7f5c1441c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:39.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62828700 1 --2- 172.21.15.73:0/974547367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f5c0b2720 0x7f7f5c1441c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:39.921 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62828700 1 -- 172.21.15.73:0/974547367 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f54009ae0 con 0x7f7f5c15af60 2023-06-23T01:08:39.922 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f62828700 1 --2- 172.21.15.73:0/974547367 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c1569e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400b610 tx=0x7f7f54018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:39.922 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f527fc700 1 -- 172.21.15.73:0/974547367 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54008ad0 con 0x7f7f5c15af60 2023-06-23T01:08:39.922 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f527fc700 1 -- 172.21.15.73:0/974547367 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f54008c30 con 0x7f7f5c15af60 2023-06-23T01:08:39.922 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f527fc700 1 -- 172.21.15.73:0/974547367 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f54025460 con 0x7f7f5c15af60 2023-06-23T01:08:39.922 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c0cf480 con 0x7f7f5c15af60 2023-06-23T01:08:39.922 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c150d40 con 0x7f7f5c15af60 2023-06-23T01:08:39.922 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f527fc700 1 -- 172.21.15.73:0/974547367 <== mon.2 v2:172.21.15.73:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7f54003040 con 0x7f7f5c15af60 2023-06-23T01:08:39.922 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.915+0000 7f7f527fc700 1 -- 172.21.15.73:0/974547367 <== mon.2 v2:172.21.15.73:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7f7f54025d40 con 0x7f7f5c15af60 2023-06-23T01:08:39.923 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.916+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c04c1b0 con 0x7f7f5c15af60 2023-06-23T01:08:39.923 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:39.918+0000 7f7f527fc700 1 -- 172.21.15.73:0/974547367 <== mon.2 v2:172.21.15.73:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f7f54020070 con 0x7f7f5c15af60 2023-06-23T01:08:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.001+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_command({"prefix": "osd new", "uuid": "e2023d49-2e44-412c-8cd1-c67447567c32", "id": 7} v 0) v1 -- 0x7f7f5c0cfc10 con 0x7f7f5c15af60 2023-06-23T01:08:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.010+0000 7f7f527fc700 1 -- 172.21.15.73:0/974547367 <== mon.2 v2:172.21.15.73:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e2023d49-2e44-412c-8cd1-c67447567c32", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7f54025890 con 0x7f7f5c15af60 2023-06-23T01:08:40.014 INFO:teuthology.orchestra.run.smithi073.stdout:7 2023-06-23T01:08:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 msgr2=0x7f7f5c1569e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 --2- 172.21.15.73:0/974547367 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c1569e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400b610 tx=0x7f7f54018000 comp rx=0 tx=0).stop 2023-06-23T01:08:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 shutdown_connections 2023-06-23T01:08:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 --2- 172.21.15.73:0/974547367 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f7f5c15af60 0x7f7f5c1569e0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 --2- 172.21.15.73:0/974547367 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f7f5c0b2720 0x7f7f5c1441c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 --2- 172.21.15.73:0/974547367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7f5c0b2af0 0x7f7f5c0b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 >> 172.21.15.73:0/974547367 conn(0x7f7f5c067450 msgr2=0x7f7f5c0ccfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 shutdown_connections 2023-06-23T01:08:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2023-06-23T01:08:40.011+0000 7f7f6428b700 1 -- 172.21.15.73:0/974547367 wait complete. 2023-06-23T01:08:40.026 INFO:tasks.ceph.osd.0:Restarting daemon 2023-06-23T01:08:40.026 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-06-23T01:08:40.028 INFO:tasks.ceph.osd.0:Started 2023-06-23T01:08:40.029 INFO:tasks.ceph.osd.1:Restarting daemon 2023-06-23T01:08:40.029 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-06-23T01:08:40.030 INFO:tasks.ceph.osd.1:Started 2023-06-23T01:08:40.031 INFO:tasks.ceph.osd.2:Restarting daemon 2023-06-23T01:08:40.031 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-06-23T01:08:40.033 INFO:tasks.ceph.osd.2:Started 2023-06-23T01:08:40.033 INFO:tasks.ceph.osd.3:Restarting daemon 2023-06-23T01:08:40.033 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-06-23T01:08:40.035 INFO:tasks.ceph.osd.3:Started 2023-06-23T01:08:40.035 INFO:tasks.ceph.osd.4:Restarting daemon 2023-06-23T01:08:40.036 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-06-23T01:08:40.039 INFO:tasks.ceph.osd.4:Started 2023-06-23T01:08:40.040 INFO:tasks.ceph.osd.5:Restarting daemon 2023-06-23T01:08:40.040 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-06-23T01:08:40.043 INFO:tasks.ceph.osd.5:Started 2023-06-23T01:08:40.044 INFO:tasks.ceph.osd.6:Restarting daemon 2023-06-23T01:08:40.044 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-06-23T01:08:40.047 INFO:tasks.ceph.osd.6:Started 2023-06-23T01:08:40.047 INFO:tasks.ceph.osd.7:Restarting daemon 2023-06-23T01:08:40.048 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-06-23T01:08:40.050 INFO:tasks.ceph.osd.7:Started 2023-06-23T01:08:40.050 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:08:40.112 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:40.108+0000 7f5e76672180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:08:40.148 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:40.145+0000 7fab8d2b3180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:08:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.190+0000 7fa12dece700 1 Processor -- start 2023-06-23T01:08:40.193 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.190+0000 7fa12dece700 1 -- start start 2023-06-23T01:08:40.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa12dece700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 0x7fa1281579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:40.194 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa12dece700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fa1280b2af0 0x7fa1280cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa12dece700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa12815c0e0 con 0x7fa12815af60 2023-06-23T01:08:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa12dece700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa12815b4d0 con 0x7fa1280b2af0 2023-06-23T01:08:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa12dece700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa12815b7b0 con 0x7fa1280b2720 2023-06-23T01:08:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa126ffd700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fa1280b2af0 0x7fa1280cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 0x7fa1281579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa126ffd700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fa1280b2af0 0x7fa1280cd6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.39:44398/0 (socket says 172.21.15.39:44398) 2023-06-23T01:08:40.195 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa126ffd700 1 -- 172.21.15.39:0/703662792 learned_addr learned my addr 172.21.15.39:0/703662792 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:40.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa126ffd700 1 -- 172.21.15.39:0/703662792 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 msgr2=0x7fa1281579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:40.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa126ffd700 1 --2- 172.21.15.39:0/703662792 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 0x7fa1281579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa126ffd700 1 -- 172.21.15.39:0/703662792 >> v1:172.21.15.73:6789/0 conn(0x7fa12815af60 legacy=0x7fa12815d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:40.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.191+0000 7fa126ffd700 1 -- 172.21.15.39:0/703662792 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa12815ba90 con 0x7fa1280b2af0 2023-06-23T01:08:40.196 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa126ffd700 1 --2- 172.21.15.39:0/703662792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1280b2af0 0x7fa1280cd6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa1100099b0 tx=0x7fa110009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28a2879e6a61fc4c server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:40.197 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:40.192+0000 7f5e76672180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:08:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa1267fc700 1 -- 172.21.15.39:0/703662792 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110009ea0 con 0x7fa1280b2af0 2023-06-23T01:08:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa1267fc700 1 -- 172.21.15.39:0/703662792 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11000b860 con 0x7fa1280b2af0 2023-06-23T01:08:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa1267fc700 1 -- 172.21.15.39:0/703662792 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa11000bc00 con 0x7fa1280b2af0 2023-06-23T01:08:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa12dece700 1 -- 172.21.15.39:0/703662792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1280b2af0 msgr2=0x7fa1280cd6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:40.199 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa12dece700 1 --2- 172.21.15.39:0/703662792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1280b2af0 0x7fa1280cd6c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fa1100099b0 tx=0x7fa110009250 comp rx=0 tx=0).stop 2023-06-23T01:08:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa12dece700 1 -- 172.21.15.39:0/703662792 shutdown_connections 2023-06-23T01:08:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa12dece700 1 --2- 172.21.15.39:0/703662792 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 0x7fa1281579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa12dece700 1 --2- 172.21.15.39:0/703662792 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa1280b2af0 0x7fa1280cd6c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa12dece700 1 -- 172.21.15.39:0/703662792 >> 172.21.15.39:0/703662792 conn(0x7fa128067450 msgr2=0x7fa128067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa12dece700 1 -- 172.21.15.39:0/703662792 shutdown_connections 2023-06-23T01:08:40.200 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.192+0000 7fa12dece700 1 -- 172.21.15.39:0/703662792 wait complete. 2023-06-23T01:08:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa12dece700 1 Processor -- start 2023-06-23T01:08:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa12dece700 1 -- start start 2023-06-23T01:08:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa12dece700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 0x7fa12814f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa12dece700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1280b2af0 0x7fa1280cff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa12dece700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa12815af60 0x7fa128153c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa12dece700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa12814f6a0 con 0x7fa1280b2af0 2023-06-23T01:08:40.201 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa12dece700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa128158a20 con 0x7fa12815af60 2023-06-23T01:08:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa12dece700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa128158d00 con 0x7fa1280b2720 2023-06-23T01:08:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa126ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1280b2af0 0x7fa1280cff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa126ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1280b2af0 0x7fa1280cff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53856/0 (socket says 172.21.15.39:53856) 2023-06-23T01:08:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa1277fe700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 0x7fa12814f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa126ffd700 1 -- 172.21.15.39:0/3004331100 learned_addr learned my addr 172.21.15.39:0/3004331100 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.194+0000 7fa127fff700 1 --2- 172.21.15.39:0/3004331100 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa12815af60 0x7fa128153c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:40.202 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa126ffd700 1 -- 172.21.15.39:0/3004331100 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 msgr2=0x7fa12814f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa126ffd700 1 --2- 172.21.15.39:0/3004331100 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 0x7fa12814f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa126ffd700 1 -- 172.21.15.39:0/3004331100 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa12815af60 msgr2=0x7fa128153c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa126ffd700 1 --2- 172.21.15.39:0/3004331100 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa12815af60 0x7fa128153c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa126ffd700 1 -- 172.21.15.39:0/3004331100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1100094f0 con 0x7fa1280b2af0 2023-06-23T01:08:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa126ffd700 1 --2- 172.21.15.39:0/3004331100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1280b2af0 0x7fa1280cff30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa128065560 tx=0x7fa110017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa11f7fe700 1 -- 172.21.15.39:0/3004331100 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110000d10 con 0x7fa1280b2af0 2023-06-23T01:08:40.203 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa11f7fe700 1 -- 172.21.15.39:0/3004331100 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa11000bbe0 con 0x7fa1280b2af0 2023-06-23T01:08:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa128158fe0 con 0x7fa1280b2af0 2023-06-23T01:08:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa11f7fe700 1 -- 172.21.15.39:0/3004331100 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa110011670 con 0x7fa1280b2af0 2023-06-23T01:08:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1280b3a10 con 0x7fa1280b2af0 2023-06-23T01:08:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa11f7fe700 1 -- 172.21.15.39:0/3004331100 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa110011aa0 con 0x7fa1280b2af0 2023-06-23T01:08:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.195+0000 7fa11f7fe700 1 -- 172.21.15.39:0/3004331100 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fa110023ad0 con 0x7fa1280b2af0 2023-06-23T01:08:40.204 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.196+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa12804c1b0 con 0x7fa1280b2af0 2023-06-23T01:08:40.205 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.198+0000 7fa11f7fe700 1 -- 172.21.15.39:0/3004331100 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fa110028070 con 0x7fa1280b2af0 2023-06-23T01:08:40.232 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:40.229+0000 7fab8d2b3180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:08:40.272 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.269+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa12815bd70 con 0x7fa1280b2af0 2023-06-23T01:08:40.273 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.270+0000 7fa11f7fe700 1 -- 172.21.15.39:0/3004331100 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fa110015020 con 0x7fa1280b2af0 2023-06-23T01:08:40.274 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:08:40.274 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":10,"fsid":"314a4b16-8059-4f2c-986c-1624c6de1afb","created":"2023-06-23T01:08:37.443723+0000","modified":"2023-06-23T01:08:40.002994+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:08:40.002994+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a33e557d-f70e-4a1c-a10e-bc7f351dee53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"757696fc-3c60-4f7f-822d-a8ff1e89534d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cffcd6d2-b414-40b0-8967-26421b058b3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5a91897e-5924-4d57-ac5e-bfd06cf4ff93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"390058ef-ef47-475e-af72-8e2f51ee184c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c34ad563-5018-4cf4-830f-2ac56a3a5e5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e2023d49-2e44-412c-8cd1-c67447567c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:08:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1280b2af0 msgr2=0x7fa1280cff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 --2- 172.21.15.39:0/3004331100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1280b2af0 0x7fa1280cff30 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa128065560 tx=0x7fa110017000 comp rx=0 tx=0).stop 2023-06-23T01:08:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 shutdown_connections 2023-06-23T01:08:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 --2- 172.21.15.39:0/3004331100 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa12815af60 0x7fa128153c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 --2- 172.21.15.39:0/3004331100 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa1280b2720 0x7fa12814f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.275 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 --2- 172.21.15.39:0/3004331100 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa1280b2af0 0x7fa1280cff30 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 >> 172.21.15.39:0/3004331100 conn(0x7fa128067450 msgr2=0x7fa1280b0900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 shutdown_connections 2023-06-23T01:08:40.276 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:40.271+0000 7fa12dece700 1 -- 172.21.15.39:0/3004331100 wait complete. 2023-06-23T01:08:40.276 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:40.271+0000 7f5e76672180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:08:40.286 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-06-23T01:08:40.286 INFO:tasks.ceph:Waiting for OSDs to come up 2023-06-23T01:08:40.314 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:40.311+0000 7fab8d2b3180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:08:40.637 INFO:tasks.ceph.osd.4.smithi073.stderr:2023-06-23T01:08:40.634+0000 7fe073825540 -1 Falling back to public interface 2023-06-23T01:08:40.638 INFO:tasks.ceph.osd.1.smithi039.stderr:2023-06-23T01:08:40.635+0000 7f373abf5540 -1 Falling back to public interface 2023-06-23T01:08:40.640 INFO:tasks.ceph.osd.6.smithi073.stderr:2023-06-23T01:08:40.637+0000 7f579473f540 -1 Falling back to public interface 2023-06-23T01:08:40.644 INFO:tasks.ceph.osd.5.smithi073.stderr:2023-06-23T01:08:40.641+0000 7f08689d4540 -1 Falling back to public interface 2023-06-23T01:08:40.649 INFO:tasks.ceph.osd.3.smithi039.stderr:2023-06-23T01:08:40.646+0000 7f546d6be540 -1 Falling back to public interface 2023-06-23T01:08:40.651 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-06-23T01:08:40.649+0000 7f901f637540 -1 Falling back to public interface 2023-06-23T01:08:40.654 INFO:tasks.ceph.osd.2.smithi039.stderr:2023-06-23T01:08:40.652+0000 7f60a083d540 -1 Falling back to public interface 2023-06-23T01:08:40.674 INFO:tasks.ceph.osd.7.smithi073.stderr:2023-06-23T01:08:40.670+0000 7ff329624540 -1 Falling back to public interface 2023-06-23T01:08:40.849 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:40.845+0000 7f5e76672180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:08:40.931 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:40.928+0000 7fab8d2b3180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:08:40.987 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:40.984+0000 7f5e76672180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:08:41.075 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:41.072+0000 7fab8d2b3180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:08:41.084 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:41.080+0000 7f5e76672180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:08:41.091 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:08:41.167 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:41.164+0000 7fab8d2b3180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:08:41.202 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.199+0000 7f80e970c700 1 Processor -- start 2023-06-23T01:08:41.202 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.199+0000 7f80e970c700 1 -- start start 2023-06-23T01:08:41.204 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e970c700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f80e41594e0 0x7f80e40b0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:41.204 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e970c700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e415b0b0 con 0x7f80e40b1250 2023-06-23T01:08:41.204 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e970c700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80e40b4b70 con 0x7f80e40b4120 2023-06-23T01:08:41.204 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e970c700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80e40b0170 con 0x7f80e41594e0 2023-06-23T01:08:41.204 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e37fe700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f80e40b1250 0x7f80e41598b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:54886/0 (socket says 172.21.15.39:54886) 2023-06-23T01:08:41.204 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e37fe700 1 -- 172.21.15.39:0/2052551123 learned_addr learned my addr 172.21.15.39:0/2052551123 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:41.205 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e2ffd700 1 --2- 172.21.15.39:0/2052551123 >> v2:172.21.15.73:3300/0 conn(0x7f80e41594e0 0x7f80e40b0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:41.205 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e1ffb700 1 -- 172.21.15.39:0/2052551123 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1557076016 0 0) 0x7f80e415b0b0 con 0x7f80e40b1250 2023-06-23T01:08:41.205 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e1ffb700 1 -- 172.21.15.39:0/2052551123 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80cc003570 con 0x7f80e40b1250 2023-06-23T01:08:41.205 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e2ffd700 1 -- 172.21.15.39:0/2052551123 >> v1:172.21.15.73:6789/0 conn(0x7f80e40b4120 legacy=0x7f80e40b0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:41.205 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e2ffd700 1 -- 172.21.15.39:0/2052551123 >> v1:172.21.15.39:6789/0 conn(0x7f80e40b1250 legacy=0x7f80e41598b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:41.205 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.200+0000 7f80e2ffd700 1 -- 172.21.15.39:0/2052551123 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80e40b45b0 con 0x7f80e41594e0 2023-06-23T01:08:41.205 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e2ffd700 1 --2- 172.21.15.39:0/2052551123 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 0x7f80e40b0600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f80d40098e0 tx=0x7f80d40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a2c7a2a536072e1 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:41.206 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e1ffb700 1 -- 172.21.15.39:0/2052551123 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d400b640 con 0x7f80e41594e0 2023-06-23T01:08:41.206 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e1ffb700 1 -- 172.21.15.39:0/2052551123 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d400b7a0 con 0x7f80e41594e0 2023-06-23T01:08:41.206 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e1ffb700 1 -- 172.21.15.39:0/2052551123 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d400bb40 con 0x7f80e41594e0 2023-06-23T01:08:41.208 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e970c700 1 -- 172.21.15.39:0/2052551123 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 msgr2=0x7f80e40b0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:41.208 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e970c700 1 --2- 172.21.15.39:0/2052551123 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 0x7f80e40b0600 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f80d40098e0 tx=0x7f80d40091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:41.208 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e970c700 1 -- 172.21.15.39:0/2052551123 shutdown_connections 2023-06-23T01:08:41.209 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e970c700 1 --2- 172.21.15.39:0/2052551123 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 0x7f80e40b0600 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:41.209 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e970c700 1 -- 172.21.15.39:0/2052551123 >> 172.21.15.39:0/2052551123 conn(0x7f80e4099740 msgr2=0x7f80e406f360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:41.209 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e970c700 1 -- 172.21.15.39:0/2052551123 shutdown_connections 2023-06-23T01:08:41.209 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.201+0000 7f80e970c700 1 -- 172.21.15.39:0/2052551123 wait complete. 2023-06-23T01:08:41.209 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e970c700 1 Processor -- start 2023-06-23T01:08:41.209 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e970c700 1 -- start start 2023-06-23T01:08:41.209 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e970c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80e40b1250 0x7f80e4156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:41.210 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e970c700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f80e40b4120 0x7f80e409b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:41.210 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e970c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 0x7f80e409d080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:41.210 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e970c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e409c820 con 0x7f80e40b1250 2023-06-23T01:08:41.210 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e970c700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80e415b5c0 con 0x7f80e41594e0 2023-06-23T01:08:41.210 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e970c700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80e4157080 con 0x7f80e40b4120 2023-06-23T01:08:41.210 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.203+0000 7f80e2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80e40b1250 0x7f80e4156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:41.210 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e2ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80e40b1250 0x7f80e4156b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:53868/0 (socket says 172.21.15.39:53868) 2023-06-23T01:08:41.210 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e2ffd700 1 -- 172.21.15.39:0/129147639 learned_addr learned my addr 172.21.15.39:0/129147639 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:41.211 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e37fe700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 0x7f80e409d080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:41.211 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e27fc700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f80e40b4120 0x7f80e409b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:41.211 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e2ffd700 1 -- 172.21.15.39:0/129147639 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f80e40b4120 msgr2=0x7f80e409b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:41.211 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e2ffd700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f80e40b4120 0x7f80e409b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:41.211 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e2ffd700 1 -- 172.21.15.39:0/129147639 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 msgr2=0x7f80e409d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:41.211 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e2ffd700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 0x7f80e409d080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:41.211 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e2ffd700 1 -- 172.21.15.39:0/129147639 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80d4009420 con 0x7f80e40b1250 2023-06-23T01:08:41.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e2ffd700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80e40b1250 0x7f80e4156b40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f80e4067f20 tx=0x7f80d4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:41.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80caffd700 1 -- 172.21.15.39:0/129147639 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d4000d10 con 0x7f80e40b1250 2023-06-23T01:08:41.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80caffd700 1 -- 172.21.15.39:0/129147639 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80d400bb20 con 0x7f80e40b1250 2023-06-23T01:08:41.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80e409c260 con 0x7f80e40b1250 2023-06-23T01:08:41.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80caffd700 1 -- 172.21.15.39:0/129147639 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d4011670 con 0x7f80e40b1250 2023-06-23T01:08:41.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.204+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80e415b250 con 0x7f80e40b1250 2023-06-23T01:08:41.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.205+0000 7f80caffd700 1 -- 172.21.15.39:0/129147639 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f80d4011c60 con 0x7f80e40b1250 2023-06-23T01:08:41.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.205+0000 7f80caffd700 1 -- 172.21.15.39:0/129147639 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f80d402d3f0 con 0x7f80e40b1250 2023-06-23T01:08:41.213 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.205+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80e415b0b0 con 0x7f80e40b1250 2023-06-23T01:08:41.213 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.207+0000 7f80caffd700 1 -- 172.21.15.39:0/129147639 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f80d40436e0 con 0x7f80e40b1250 2023-06-23T01:08:41.282 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.279+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80e404c1b0 con 0x7f80e40b1250 2023-06-23T01:08:41.283 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.280+0000 7f80caffd700 1 -- 172.21.15.39:0/129147639 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f80d4032070 con 0x7f80e40b1250 2023-06-23T01:08:41.283 INFO:teuthology.misc.health.smithi039.stdout: 2023-06-23T01:08:41.283 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":10,"fsid":"314a4b16-8059-4f2c-986c-1624c6de1afb","created":"2023-06-23T01:08:37.443723+0000","modified":"2023-06-23T01:08:40.002994+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:08:40.002994+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a33e557d-f70e-4a1c-a10e-bc7f351dee53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"757696fc-3c60-4f7f-822d-a8ff1e89534d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cffcd6d2-b414-40b0-8967-26421b058b3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5a91897e-5924-4d57-ac5e-bfd06cf4ff93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"390058ef-ef47-475e-af72-8e2f51ee184c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c34ad563-5018-4cf4-830f-2ac56a3a5e5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e2023d49-2e44-412c-8cd1-c67447567c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:08:41.284 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.280+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80e40b1250 msgr2=0x7f80e4156b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:41.284 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.280+0000 7f80e970c700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80e40b1250 0x7f80e4156b40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f80e4067f20 tx=0x7f80d4017000 comp rx=0 tx=0).stop 2023-06-23T01:08:41.284 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.281+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 shutdown_connections 2023-06-23T01:08:41.285 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.281+0000 7f80e970c700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80e41594e0 0x7f80e409d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:41.285 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.281+0000 7f80e970c700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f80e40b1250 0x7f80e4156b40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:41.285 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.281+0000 7f80e970c700 1 --2- 172.21.15.39:0/129147639 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f80e40b4120 0x7f80e409b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:41.285 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.281+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 >> 172.21.15.39:0/129147639 conn(0x7f80e4099740 msgr2=0x7f80e41588e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:41.285 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.281+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 shutdown_connections 2023-06-23T01:08:41.285 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:41.281+0000 7f80e970c700 1 -- 172.21.15.39:0/129147639 wait complete. 2023-06-23T01:08:41.295 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-23T01:08:41.332 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:41.328+0000 7f5e76672180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:08:41.407 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:41.404+0000 7f5e76672180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:08:41.428 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:41.425+0000 7fab8d2b3180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:08:41.510 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:41.507+0000 7fab8d2b3180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:08:42.230 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.227+0000 7f5e76672180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:08:42.305 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.302+0000 7f5e76672180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:08:42.380 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.376+0000 7f5e76672180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:08:42.399 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:42.396+0000 7fab8d2b3180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:08:42.477 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:42.474+0000 7fab8d2b3180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:08:42.511 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.508+0000 7f5e76672180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:08:42.555 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:42.552+0000 7fab8d2b3180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:08:42.598 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.595+0000 7f5e76672180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:08:42.695 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:42.692+0000 7fab8d2b3180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:08:42.748 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.745+0000 7f5e76672180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:08:42.784 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:42.781+0000 7fab8d2b3180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:08:42.824 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.821+0000 7f5e76672180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:08:42.900 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.897+0000 7f5e76672180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:08:42.942 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:42.939+0000 7fab8d2b3180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:08:42.976 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:42.973+0000 7f5e76672180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:08:43.021 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:43.018+0000 7fab8d2b3180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:08:43.101 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:43.098+0000 7fab8d2b3180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:08:43.182 INFO:tasks.ceph.mgr.y.smithi039.stderr:2023-06-23T01:08:43.179+0000 7fab8d2b3180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:08:44.002 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:43.999+0000 7f5e2fa76700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-06-23T01:08:44.002 INFO:tasks.ceph.mgr.x.smithi073.stderr:2023-06-23T01:08:43.999+0000 7f5e2fa76700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-06-23T01:08:44.130 INFO:tasks.ceph.osd.3.smithi039.stderr:2023-06-23T01:08:44.127+0000 7f546d6be540 -1 osd.3 0 log_to_monitors true 2023-06-23T01:08:44.137 INFO:tasks.ceph.osd.4.smithi073.stderr:2023-06-23T01:08:44.133+0000 7fe073825540 -1 osd.4 0 log_to_monitors true 2023-06-23T01:08:44.142 INFO:tasks.ceph.osd.6.smithi073.stderr:2023-06-23T01:08:44.139+0000 7f579473f540 -1 osd.6 0 log_to_monitors true 2023-06-23T01:08:44.144 INFO:tasks.ceph.osd.1.smithi039.stderr:2023-06-23T01:08:44.141+0000 7f373abf5540 -1 osd.1 0 log_to_monitors true 2023-06-23T01:08:44.147 INFO:tasks.ceph.osd.5.smithi073.stderr:2023-06-23T01:08:44.144+0000 7f08689d4540 -1 osd.5 0 log_to_monitors true 2023-06-23T01:08:44.174 INFO:tasks.ceph.osd.2.smithi039.stderr:2023-06-23T01:08:44.171+0000 7f60a083d540 -1 osd.2 0 log_to_monitors true 2023-06-23T01:08:44.174 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-06-23T01:08:44.171+0000 7f901f637540 -1 osd.0 0 log_to_monitors true 2023-06-23T01:08:44.178 INFO:tasks.ceph.osd.7.smithi073.stderr:2023-06-23T01:08:44.175+0000 7ff329624540 -1 osd.7 0 log_to_monitors true 2023-06-23T01:08:47.489 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-06-23T01:08:47.486+0000 7f9016259700 -1 osd.0 0 waiting for initial osdmap 2023-06-23T01:08:47.493 INFO:tasks.ceph.osd.0.smithi039.stderr:2023-06-23T01:08:47.490+0000 7f900f050700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:08:47.496 INFO:tasks.ceph.osd.2.smithi039.stderr:2023-06-23T01:08:47.494+0000 7f609745f700 -1 osd.2 0 waiting for initial osdmap 2023-06-23T01:08:47.501 INFO:tasks.ceph.osd.2.smithi039.stderr:2023-06-23T01:08:47.499+0000 7f6090256700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:08:47.528 INFO:tasks.ceph.osd.1.smithi039.stderr:2023-06-23T01:08:47.525+0000 7f3731817700 -1 osd.1 0 waiting for initial osdmap 2023-06-23T01:08:47.531 INFO:tasks.ceph.osd.1.smithi039.stderr:2023-06-23T01:08:47.529+0000 7f372a60e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:08:47.610 INFO:tasks.ceph.osd.3.smithi039.stderr:2023-06-23T01:08:47.607+0000 7f54642e0700 -1 osd.3 0 waiting for initial osdmap 2023-06-23T01:08:47.615 INFO:tasks.ceph.osd.3.smithi039.stderr:2023-06-23T01:08:47.612+0000 7f545d0d7700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:08:48.049 INFO:tasks.ceph.osd.6.smithi073.stderr:2023-06-23T01:08:48.047+0000 7f578b361700 -1 osd.6 0 waiting for initial osdmap 2023-06-23T01:08:48.057 INFO:tasks.ceph.osd.7.smithi073.stderr:2023-06-23T01:08:48.054+0000 7ff320246700 -1 osd.7 0 waiting for initial osdmap 2023-06-23T01:08:48.060 INFO:tasks.ceph.osd.6.smithi073.stderr:2023-06-23T01:08:48.058+0000 7f5784158700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:08:48.065 INFO:tasks.ceph.osd.5.smithi073.stderr:2023-06-23T01:08:48.062+0000 7f085f5f6700 -1 osd.5 0 waiting for initial osdmap 2023-06-23T01:08:48.065 INFO:tasks.ceph.osd.7.smithi073.stderr:2023-06-23T01:08:48.063+0000 7ff31903d700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:08:48.069 INFO:tasks.ceph.osd.5.smithi073.stderr:2023-06-23T01:08:48.066+0000 7f08583ed700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:08:48.090 INFO:tasks.ceph.osd.4.smithi073.stderr:2023-06-23T01:08:48.088+0000 7fe06a447700 -1 osd.4 0 waiting for initial osdmap 2023-06-23T01:08:48.098 INFO:tasks.ceph.osd.4.smithi073.stderr:2023-06-23T01:08:48.096+0000 7fe06323e700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:08:48.100 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:08:48.211 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.208+0000 7fb638905700 1 Processor -- start 2023-06-23T01:08:48.212 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.208+0000 7fb638905700 1 -- start start 2023-06-23T01:08:48.218 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb638905700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7fb6340b5ef0 0x7fb6340ae4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:48.218 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb638905700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb63415a6e0 0x7fb6340afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:48.218 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb638905700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb63415a250 con 0x7fb634153aa0 2023-06-23T01:08:48.218 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb638905700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6340d0e30 con 0x7fb63415a6e0 2023-06-23T01:08:48.219 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb638905700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6340b1af0 con 0x7fb6340b5ef0 2023-06-23T01:08:48.219 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb633fff700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb63415a6e0 0x7fb6340afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:48.219 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb633fff700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fb63415a6e0 0x7fb6340afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56586/0 (socket says 172.21.15.39:56586) 2023-06-23T01:08:48.220 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb633fff700 1 -- 172.21.15.39:0/1322795605 learned_addr learned my addr 172.21.15.39:0/1322795605 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:48.220 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.209+0000 7fb632ffd700 1 --2- 172.21.15.39:0/1322795605 >> v2:172.21.15.73:3300/0 conn(0x7fb6340b5ef0 0x7fb6340ae4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:48.220 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb633fff700 1 -- 172.21.15.39:0/1322795605 >> v2:172.21.15.73:3300/0 conn(0x7fb6340b5ef0 msgr2=0x7fb6340ae4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:48.220 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb633fff700 1 --2- 172.21.15.39:0/1322795605 >> v2:172.21.15.73:3300/0 conn(0x7fb6340b5ef0 0x7fb6340ae4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.220 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb633fff700 1 -- 172.21.15.39:0/1322795605 >> v1:172.21.15.73:6789/0 conn(0x7fb634153aa0 legacy=0x7fb6340adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:48.221 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb633fff700 1 -- 172.21.15.39:0/1322795605 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6340d0b50 con 0x7fb63415a6e0 2023-06-23T01:08:48.221 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb6327fc700 1 -- 172.21.15.39:0/1322795605 <== mon.1 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3170481657 0 0) 0x7fb63415a250 con 0x7fb634153aa0 2023-06-23T01:08:48.221 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb633fff700 1 --2- 172.21.15.39:0/1322795605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb63415a6e0 0x7fb6340afb40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb63408a570 tx=0x7fb62c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd67c7d61b59ba1 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:48.221 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb6327fc700 1 -- 172.21.15.39:0/1322795605 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c00e070 con 0x7fb63415a6e0 2023-06-23T01:08:48.221 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb6327fc700 1 -- 172.21.15.39:0/1322795605 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62c00ba20 con 0x7fb63415a6e0 2023-06-23T01:08:48.222 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb6327fc700 1 -- 172.21.15.39:0/1322795605 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62c0092e0 con 0x7fb63415a6e0 2023-06-23T01:08:48.222 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.210+0000 7fb638905700 1 -- 172.21.15.39:0/1322795605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb63415a6e0 msgr2=0x7fb6340afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:48.222 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.211+0000 7fb638905700 1 --2- 172.21.15.39:0/1322795605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb63415a6e0 0x7fb6340afb40 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fb63408a570 tx=0x7fb62c009840 comp rx=0 tx=0).stop 2023-06-23T01:08:48.222 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.211+0000 7fb638905700 1 -- 172.21.15.39:0/1322795605 shutdown_connections 2023-06-23T01:08:48.223 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.211+0000 7fb638905700 1 --2- 172.21.15.39:0/1322795605 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb63415a6e0 0x7fb6340afb40 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.223 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.211+0000 7fb638905700 1 --2- 172.21.15.39:0/1322795605 >> v2:172.21.15.73:3300/0 conn(0x7fb6340b5ef0 0x7fb6340ae4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.223 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.211+0000 7fb638905700 1 -- 172.21.15.39:0/1322795605 >> 172.21.15.39:0/1322795605 conn(0x7fb63406f360 msgr2=0x7fb6340d02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:48.223 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.211+0000 7fb638905700 1 -- 172.21.15.39:0/1322795605 shutdown_connections 2023-06-23T01:08:48.224 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.211+0000 7fb638905700 1 -- 172.21.15.39:0/1322795605 wait complete. 2023-06-23T01:08:48.224 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.212+0000 7fb638905700 1 Processor -- start 2023-06-23T01:08:48.224 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.212+0000 7fb638905700 1 -- start start 2023-06-23T01:08:48.224 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb638905700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb6340b5ef0 0x7fb6340b3300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:48.224 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb638905700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb634153aa0 0x7fb63414adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:48.225 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb638905700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb63415a6e0 0x7fb6340b4e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:48.225 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb638905700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340b4990 con 0x7fb634153aa0 2023-06-23T01:08:48.225 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb638905700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340b3d80 con 0x7fb63415a6e0 2023-06-23T01:08:48.225 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb638905700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6340b4060 con 0x7fb6340b5ef0 2023-06-23T01:08:48.225 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb634153aa0 0x7fb63414adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:48.226 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb634153aa0 0x7fb63414adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56594/0 (socket says 172.21.15.39:56594) 2023-06-23T01:08:48.226 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 -- 172.21.15.39:0/1103271895 learned_addr learned my addr 172.21.15.39:0/1103271895 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:48.226 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb6337fe700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb6340b5ef0 0x7fb6340b3300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:48.226 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb633fff700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb63415a6e0 0x7fb6340b4e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:48.227 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 -- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb6340b5ef0 msgr2=0x7fb6340b3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:48.227 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb6340b5ef0 0x7fb6340b3300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.227 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 -- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb63415a6e0 msgr2=0x7fb6340b4e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:48.227 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb63415a6e0 0x7fb6340b4e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.228 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 -- 172.21.15.39:0/1103271895 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c009ae0 con 0x7fb634153aa0 2023-06-23T01:08:48.228 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb632ffd700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb634153aa0 0x7fb63414adf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb62800e930 tx=0x7fb6280091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:48.228 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb61b7fe700 1 -- 172.21.15.39:0/1103271895 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628013680 con 0x7fb634153aa0 2023-06-23T01:08:48.228 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb61b7fe700 1 -- 172.21.15.39:0/1103271895 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6280137e0 con 0x7fb634153aa0 2023-06-23T01:08:48.228 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6340b4340 con 0x7fb634153aa0 2023-06-23T01:08:48.229 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb61b7fe700 1 -- 172.21.15.39:0/1103271895 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628013b80 con 0x7fb634153aa0 2023-06-23T01:08:48.229 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.213+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6340b3840 con 0x7fb634153aa0 2023-06-23T01:08:48.229 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.214+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6340b4b00 con 0x7fb634153aa0 2023-06-23T01:08:48.229 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.215+0000 7fb61b7fe700 1 -- 172.21.15.39:0/1103271895 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7fb62800c3e0 con 0x7fb634153aa0 2023-06-23T01:08:48.230 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.215+0000 7fb61b7fe700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fb620066e00 0x7fb6200692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:48.230 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.215+0000 7fb61b7fe700 1 -- 172.21.15.39:0/1103271895 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7fb628083660 con 0x7fb634153aa0 2023-06-23T01:08:48.231 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.215+0000 7fb6337fe700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fb620066e00 0x7fb6200692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:48.231 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.216+0000 7fb6337fe700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fb620066e00 0x7fb6200692b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb624007fd0 tx=0x7fb624007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:48.231 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.220+0000 7fb61b7fe700 1 -- 172.21.15.39:0/1103271895 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fb628051800 con 0x7fb634153aa0 2023-06-23T01:08:48.419 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.416+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6340b3ad0 con 0x7fb634153aa0 2023-06-23T01:08:48.420 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.416+0000 7fb61b7fe700 1 -- 172.21.15.39:0/1103271895 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fb628054e20 con 0x7fb634153aa0 2023-06-23T01:08:48.420 INFO:teuthology.misc.health.smithi039.stdout: 2023-06-23T01:08:48.420 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":12,"fsid":"314a4b16-8059-4f2c-986c-1624c6de1afb","created":"2023-06-23T01:08:37.443723+0000","modified":"2023-06-23T01:08:45.996234+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:08:40.002994+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"a33e557d-f70e-4a1c-a10e-bc7f351dee53","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"757696fc-3c60-4f7f-822d-a8ff1e89534d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cffcd6d2-b414-40b0-8967-26421b058b3e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5a91897e-5924-4d57-ac5e-bfd06cf4ff93","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"390058ef-ef47-475e-af72-8e2f51ee184c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"c34ad563-5018-4cf4-830f-2ac56a3a5e5f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"e2023d49-2e44-412c-8cd1-c67447567c32","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:08:48.422 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fb620066e00 msgr2=0x7fb6200692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:48.422 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fb620066e00 0x7fb6200692b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb624007fd0 tx=0x7fb624007d30 comp rx=0 tx=0).stop 2023-06-23T01:08:48.422 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb634153aa0 msgr2=0x7fb63414adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:48.422 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb634153aa0 0x7fb63414adf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb62800e930 tx=0x7fb6280091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.422 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 shutdown_connections 2023-06-23T01:08:48.422 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fb620066e00 0x7fb6200692b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.422 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb63415a6e0 0x7fb6340b4e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.423 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fb6340b5ef0 0x7fb6340b3300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.423 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 --2- 172.21.15.39:0/1103271895 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fb634153aa0 0x7fb63414adf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:48.423 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 >> 172.21.15.39:0/1103271895 conn(0x7fb63406f360 msgr2=0x7fb63415c390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:48.423 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.418+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 shutdown_connections 2023-06-23T01:08:48.423 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:48.419+0000 7fb638905700 1 -- 172.21.15.39:0/1103271895 wait complete. 2023-06-23T01:08:48.432 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-23T01:08:55.238 DEBUG:teuthology.orchestra.run.smithi039:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:08:55.358 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.354+0000 7f41df380700 1 Processor -- start 2023-06-23T01:08:55.359 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.354+0000 7f41df380700 1 -- start start 2023-06-23T01:08:55.359 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.355+0000 7f41df380700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f41d80b06c0 0x7f41d80b0b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.360 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.355+0000 7f41df380700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d806c3e0 con 0x7f41d8159be0 2023-06-23T01:08:55.360 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.355+0000 7f41df380700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41d815a7f0 con 0x7f41d8153b60 2023-06-23T01:08:55.360 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.355+0000 7f41df380700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41d815aad0 con 0x7f41d80b06c0 2023-06-23T01:08:55.360 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.355+0000 7f41dc91b700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f41d80b06c0 0x7f41d80b0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.360 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41dd91d700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f41d8159be0 0x7f41d815a020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:42458/0 (socket says 172.21.15.39:42458) 2023-06-23T01:08:55.360 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41dc91b700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f41d80b06c0 0x7f41d80b0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56604/0 (socket says 172.21.15.39:56604) 2023-06-23T01:08:55.361 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41dd91d700 1 -- 172.21.15.39:0/3085109054 learned_addr learned my addr 172.21.15.39:0/3085109054 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:55.362 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41dc91b700 1 -- 172.21.15.39:0/3085109054 >> v1:172.21.15.73:6789/0 conn(0x7f41d8153b60 legacy=0x7f41d8067450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.362 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41d7fff700 1 -- 172.21.15.39:0/3085109054 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985247176 0 0) 0x7f41d806c3e0 con 0x7f41d8159be0 2023-06-23T01:08:55.362 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41dc91b700 1 -- 172.21.15.39:0/3085109054 >> v1:172.21.15.39:6789/0 conn(0x7f41d8159be0 legacy=0x7f41d815a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.362 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41dc91b700 1 -- 172.21.15.39:0/3085109054 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41d80afbe0 con 0x7f41d80b06c0 2023-06-23T01:08:55.363 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41dc91b700 1 --2- 172.21.15.39:0/3085109054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d80b0b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f41cc009860 tx=0x7f41cc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5ca8e9c338b1cff server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:55.363 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41d7fff700 1 -- 172.21.15.39:0/3085109054 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc011630 con 0x7f41d80b06c0 2023-06-23T01:08:55.363 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41d7fff700 1 -- 172.21.15.39:0/3085109054 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41cc011790 con 0x7f41d80b06c0 2023-06-23T01:08:55.363 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.356+0000 7f41d7fff700 1 -- 172.21.15.39:0/3085109054 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41cc011b50 con 0x7f41d80b06c0 2023-06-23T01:08:55.363 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.357+0000 7f41df380700 1 -- 172.21.15.39:0/3085109054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 msgr2=0x7f41d80b0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.363 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.357+0000 7f41df380700 1 --2- 172.21.15.39:0/3085109054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d80b0b20 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f41cc009860 tx=0x7f41cc0091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.363 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.357+0000 7f41df380700 1 -- 172.21.15.39:0/3085109054 shutdown_connections 2023-06-23T01:08:55.363 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.357+0000 7f41df380700 1 --2- 172.21.15.39:0/3085109054 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d80b0b20 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.364 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.357+0000 7f41df380700 1 -- 172.21.15.39:0/3085109054 >> 172.21.15.39:0/3085109054 conn(0x7f41d80c9310 msgr2=0x7f41d80c9710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:55.364 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.357+0000 7f41df380700 1 -- 172.21.15.39:0/3085109054 shutdown_connections 2023-06-23T01:08:55.364 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.357+0000 7f41df380700 1 -- 172.21.15.39:0/3085109054 wait complete. 2023-06-23T01:08:55.364 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.358+0000 7f41df380700 1 Processor -- start 2023-06-23T01:08:55.364 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.358+0000 7f41df380700 1 -- start start 2023-06-23T01:08:55.366 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41df380700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d815e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.367 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41df380700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f41d8153b60 0x7f41d816ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.367 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41df380700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41d8159be0 0x7f41d8157b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.367 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41df380700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d816ab90 con 0x7f41d80b06c0 2023-06-23T01:08:55.367 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41df380700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41d8156050 con 0x7f41d8159be0 2023-06-23T01:08:55.367 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41df380700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41d815eba0 con 0x7f41d8153b60 2023-06-23T01:08:55.367 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd11c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d815e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.367 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd11c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d815e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56620/0 (socket says 172.21.15.39:56620) 2023-06-23T01:08:55.368 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd11c700 1 -- 172.21.15.39:0/3052957130 learned_addr learned my addr 172.21.15.39:0/3052957130 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:55.368 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd91d700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41d8159be0 0x7f41d8157b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.368 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dc91b700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f41d8153b60 0x7f41d816ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.368 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd11c700 1 -- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f41d8153b60 msgr2=0x7f41d816ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.368 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd11c700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f41d8153b60 0x7f41d816ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.368 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd11c700 1 -- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41d8159be0 msgr2=0x7f41d8157b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.368 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd11c700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41d8159be0 0x7f41d8157b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.368 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd11c700 1 -- 172.21.15.39:0/3052957130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41cc009420 con 0x7f41d80b06c0 2023-06-23T01:08:55.369 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dd91d700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41d8159be0 0x7f41d8157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:08:55.369 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.359+0000 7f41dc91b700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f41d8153b60 0x7f41d816ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:08:55.369 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.360+0000 7f41dd11c700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d815e660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41c800cb90 tx=0x7f41c80018e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:55.369 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.360+0000 7f41d57fa700 1 -- 172.21.15.39:0/3052957130 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c800a870 con 0x7f41d80b06c0 2023-06-23T01:08:55.369 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.360+0000 7f41d57fa700 1 -- 172.21.15.39:0/3052957130 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41c800a9d0 con 0x7f41d80b06c0 2023-06-23T01:08:55.369 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.360+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41d816a8b0 con 0x7f41d80b06c0 2023-06-23T01:08:55.369 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.360+0000 7f41d57fa700 1 -- 172.21.15.39:0/3052957130 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41c800ad90 con 0x7f41d80b06c0 2023-06-23T01:08:55.370 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.360+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41d8155ce0 con 0x7f41d80b06c0 2023-06-23T01:08:55.370 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.361+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41d804c1b0 con 0x7f41d80b06c0 2023-06-23T01:08:55.370 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.361+0000 7f41d57fa700 1 -- 172.21.15.39:0/3052957130 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f41c8012630 con 0x7f41d80b06c0 2023-06-23T01:08:55.370 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.361+0000 7f41d57fa700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f41c4066e90 0x7f41c4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.370 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.362+0000 7f41d57fa700 1 -- 172.21.15.39:0/3052957130 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f41c8084660 con 0x7f41d80b06c0 2023-06-23T01:08:55.370 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.362+0000 7f41dc91b700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f41c4066e90 0x7f41c4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.370 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.362+0000 7f41dc91b700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f41c4066e90 0x7f41c4069340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0093f0 tx=0x7f41cc016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:55.371 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.366+0000 7f41d57fa700 1 -- 172.21.15.39:0/3052957130 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f41c8051ce0 con 0x7f41d80b06c0 2023-06-23T01:08:55.566 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.562+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f41d8070660 con 0x7f41d80b06c0 2023-06-23T01:08:55.566 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.562+0000 7f41d57fa700 1 -- 172.21.15.39:0/3052957130 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12175 (secure 0 0 0) 0x7f41c8055300 con 0x7f41d80b06c0 2023-06-23T01:08:55.567 INFO:teuthology.misc.health.smithi039.stdout: 2023-06-23T01:08:55.567 INFO:teuthology.misc.health.smithi039.stdout:{"epoch":16,"fsid":"314a4b16-8059-4f2c-986c-1624c6de1afb","created":"2023-06-23T01:08:37.443723+0000","modified":"2023-06-23T01:08:51.505705+0000","last_up_change":"2023-06-23T01:08:48.497542+0000","last_in_change":"2023-06-23T01:08:40.002994+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:08:49.048645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"a33e557d-f70e-4a1c-a10e-bc7f351dee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":117964},{"type":"v1","addr":"172.21.15.39:6817","nonce":117964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":117964},{"type":"v1","addr":"172.21.15.39:6819","nonce":117964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":117964},{"type":"v1","addr":"172.21.15.39:6823","nonce":117964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":117964},{"type":"v1","addr":"172.21.15.39:6821","nonce":117964}]},"public_addr":"172.21.15.39:6817/117964","cluster_addr":"172.21.15.39:6819/117964","heartbeat_back_addr":"172.21.15.39:6823/117964","heartbeat_front_addr":"172.21.15.39:6821/117964","state":["exists","up"]},{"osd":1,"uuid":"757696fc-3c60-4f7f-822d-a8ff1e89534d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":117963},{"type":"v1","addr":"172.21.15.39:6801","nonce":117963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":117963},{"type":"v1","addr":"172.21.15.39:6803","nonce":117963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":117963},{"type":"v1","addr":"172.21.15.39:6807","nonce":117963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":117963},{"type":"v1","addr":"172.21.15.39:6805","nonce":117963}]},"public_addr":"172.21.15.39:6801/117963","cluster_addr":"172.21.15.39:6803/117963","heartbeat_back_addr":"172.21.15.39:6807/117963","heartbeat_front_addr":"172.21.15.39:6805/117963","state":["exists","up"]},{"osd":2,"uuid":"cffcd6d2-b414-40b0-8967-26421b058b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":117974},{"type":"v1","addr":"172.21.15.39:6825","nonce":117974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":117974},{"type":"v1","addr":"172.21.15.39:6827","nonce":117974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":117974},{"type":"v1","addr":"172.21.15.39:6831","nonce":117974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":117974},{"type":"v1","addr":"172.21.15.39:6829","nonce":117974}]},"public_addr":"172.21.15.39:6825/117974","cluster_addr":"172.21.15.39:6827/117974","heartbeat_back_addr":"172.21.15.39:6831/117974","heartbeat_front_addr":"172.21.15.39:6829/117974","state":["exists","up"]},{"osd":3,"uuid":"5a91897e-5924-4d57-ac5e-bfd06cf4ff93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":117973},{"type":"v1","addr":"172.21.15.39:6809","nonce":117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":117973},{"type":"v1","addr":"172.21.15.39:6811","nonce":117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":117973},{"type":"v1","addr":"172.21.15.39:6815","nonce":117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":117973},{"type":"v1","addr":"172.21.15.39:6813","nonce":117973}]},"public_addr":"172.21.15.39:6809/117973","cluster_addr":"172.21.15.39:6811/117973","heartbeat_back_addr":"172.21.15.39:6815/117973","heartbeat_front_addr":"172.21.15.39:6813/117973","state":["exists","up"]},{"osd":4,"uuid":"67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":118146},{"type":"v1","addr":"172.21.15.73:6801","nonce":118146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":118146},{"type":"v1","addr":"172.21.15.73:6803","nonce":118146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":118146},{"type":"v1","addr":"172.21.15.73:6807","nonce":118146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":118146},{"type":"v1","addr":"172.21.15.73:6805","nonce":118146}]},"public_addr":"172.21.15.73:6801/118146","cluster_addr":"172.21.15.73:6803/118146","heartbeat_back_addr":"172.21.15.73:6807/118146","heartbeat_front_addr":"172.21.15.73:6805/118146","state":["exists","up"]},{"osd":5,"uuid":"390058ef-ef47-475e-af72-8e2f51ee184c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":118148},{"type":"v1","addr":"172.21.15.73:6817","nonce":118148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":118148},{"type":"v1","addr":"172.21.15.73:6819","nonce":118148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":118148},{"type":"v1","addr":"172.21.15.73:6823","nonce":118148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":118148},{"type":"v1","addr":"172.21.15.73:6821","nonce":118148}]},"public_addr":"172.21.15.73:6817/118148","cluster_addr":"172.21.15.73:6819/118148","heartbeat_back_addr":"172.21.15.73:6823/118148","heartbeat_front_addr":"172.21.15.73:6821/118148","state":["exists","up"]},{"osd":6,"uuid":"c34ad563-5018-4cf4-830f-2ac56a3a5e5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":118147},{"type":"v1","addr":"172.21.15.73:6809","nonce":118147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":118147},{"type":"v1","addr":"172.21.15.73:6811","nonce":118147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":118147},{"type":"v1","addr":"172.21.15.73:6815","nonce":118147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":118147},{"type":"v1","addr":"172.21.15.73:6813","nonce":118147}]},"public_addr":"172.21.15.73:6809/118147","cluster_addr":"172.21.15.73:6811/118147","heartbeat_back_addr":"172.21.15.73:6815/118147","heartbeat_front_addr":"172.21.15.73:6813/118147","state":["exists","up"]},{"osd":7,"uuid":"e2023d49-2e44-412c-8cd1-c67447567c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":118189},{"type":"v1","addr":"172.21.15.73:6825","nonce":118189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":118189},{"type":"v1","addr":"172.21.15.73:6827","nonce":118189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":118189},{"type":"v1","addr":"172.21.15.73:6831","nonce":118189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":118189},{"type":"v1","addr":"172.21.15.73:6829","nonce":118189}]},"public_addr":"172.21.15.73:6825/118189","cluster_addr":"172.21.15.73:6827/118189","heartbeat_back_addr":"172.21.15.73:6831/118189","heartbeat_front_addr":"172.21.15.73:6829/118189","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.198991+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.140721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.167693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.101533+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.095800+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.189388+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.133372+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.190656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:08:55.569 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f41c4066e90 msgr2=0x7f41c4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.569 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f41c4066e90 0x7f41c4069340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f41cc0093f0 tx=0x7f41cc016000 comp rx=0 tx=0).stop 2023-06-23T01:08:55.569 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 msgr2=0x7f41d815e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.569 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d815e660 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f41c800cb90 tx=0x7f41c80018e0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.570 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 shutdown_connections 2023-06-23T01:08:55.570 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f41c4066e90 0x7f41c4069340 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.570 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41d8159be0 0x7f41d8157b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.570 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f41d80b06c0 0x7f41d815e660 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.570 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 --2- 172.21.15.39:0/3052957130 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f41d8153b60 0x7f41d816ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.570 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 >> 172.21.15.39:0/3052957130 conn(0x7f41d80c9310 msgr2=0x7f41d81530c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:55.570 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 shutdown_connections 2023-06-23T01:08:55.570 INFO:teuthology.misc.health.smithi039.stderr:2023-06-23T01:08:55.565+0000 7f41df380700 1 -- 172.21.15.39:0/3052957130 wait complete. 2023-06-23T01:08:55.580 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-06-23T01:08:55.581 INFO:tasks.ceph:Creating RBD pool 2023-06-23T01:08:55.581 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd pool create rbd 8 2023-06-23T01:08:55.699 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.695+0000 7f19c8729700 1 Processor -- start 2023-06-23T01:08:55.699 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.695+0000 7f19c8729700 1 -- start start 2023-06-23T01:08:55.703 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c8729700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f19c00b2720 0x7f19c01579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.703 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c8729700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c00b2af0 0x7f19c00cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.703 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c8729700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19c015c0e0 con 0x7f19c015af60 2023-06-23T01:08:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c8729700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19c015b4d0 con 0x7f19c00b2720 2023-06-23T01:08:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c8729700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015b7b0 con 0x7f19c00b2af0 2023-06-23T01:08:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c6cc6700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f19c015af60 0x7f19c015d390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:42462/0 (socket says 172.21.15.39:42462) 2023-06-23T01:08:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c6cc6700 1 -- 172.21.15.39:0/1995949691 learned_addr learned my addr 172.21.15.39:0/1995949691 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c64c5700 1 --2- 172.21.15.39:0/1995949691 >> v2:172.21.15.73:3300/0 conn(0x7f19c00b2720 0x7f19c01579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c5cc4700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c00b2af0 0x7f19c00cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711943176 0 0) 0x7f19c015c0e0 con 0x7f19c015af60 2023-06-23T01:08:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19ac003570 con 0x7f19c015af60 2023-06-23T01:08:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1024195363 0 0) 0x7f19ac003570 con 0x7f19c015af60 2023-06-23T01:08:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c00b2af0 msgr2=0x7f19c00cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c00b2af0 0x7f19c00cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 >> v2:172.21.15.73:3300/0 conn(0x7f19c00b2720 msgr2=0x7f19c01579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 --2- 172.21.15.39:0/1995949691 >> v2:172.21.15.73:3300/0 conn(0x7f19c00b2720 0x7f19c01579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c015ba90 con 0x7f19c015af60 2023-06-23T01:08:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 979813403 0 0) 0x7f19b80017f0 con 0x7f19c015af60 2023-06-23T01:08:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 >> v1:172.21.15.39:6789/0 conn(0x7f19c015af60 legacy=0x7f19c015d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c00b2af0 0x7f19ac003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19ac003fe0 0x7f19ac004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac004d50 0x7f19ac0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.707 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015b4d0 con 0x7f19ac003fe0 2023-06-23T01:08:55.707 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c015b7b0 con 0x7f19ac004d50 2023-06-23T01:08:55.707 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19ac00ba90 con 0x7f19c00b2af0 2023-06-23T01:08:55.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.696+0000 7f19c5cc4700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19ac003fe0 0x7f19ac004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c64c5700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c00b2af0 0x7f19ac003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c6cc6700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac004d50 0x7f19ac0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c5cc4700 1 -- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c00b2af0 msgr2=0x7f19ac003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:55.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c5cc4700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c00b2af0 0x7f19ac003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c5cc4700 1 -- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac004d50 msgr2=0x7f19ac0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c5cc4700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac004d50 0x7f19ac0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c5cc4700 1 -- 172.21.15.39:0/1995949691 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19ac003570 con 0x7f19ac003fe0 2023-06-23T01:08:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c5cc4700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19ac003fe0 0x7f19ac004810 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009c00 tx=0x7f19b00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f25871144f71f826 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b000e6b0 con 0x7f19ac003fe0 2023-06-23T01:08:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b000e810 con 0x7f19ac003fe0 2023-06-23T01:08:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.697+0000 7f19c54c3700 1 -- 172.21.15.39:0/1995949691 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b000ebd0 con 0x7f19ac003fe0 2023-06-23T01:08:55.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.698+0000 7f19c8729700 1 -- 172.21.15.39:0/1995949691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19ac003fe0 msgr2=0x7f19ac004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.698+0000 7f19c8729700 1 --2- 172.21.15.39:0/1995949691 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19ac003fe0 0x7f19ac004810 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f19b0009c00 tx=0x7f19b00091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.698+0000 7f19c6cc6700 1 -- 172.21.15.39:0/1995949691 reap_dead start 2023-06-23T01:08:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.698+0000 7f19c8729700 1 -- 172.21.15.39:0/1995949691 shutdown_connections 2023-06-23T01:08:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.698+0000 7f19c8729700 1 -- 172.21.15.39:0/1995949691 >> 172.21.15.39:0/1995949691 conn(0x7f19c0067450 msgr2=0x7f19c0067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.698+0000 7f19c8729700 1 -- 172.21.15.39:0/1995949691 shutdown_connections 2023-06-23T01:08:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.698+0000 7f19c8729700 1 -- 172.21.15.39:0/1995949691 wait complete. 2023-06-23T01:08:55.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.699+0000 7f19c8729700 1 Processor -- start 2023-06-23T01:08:55.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.699+0000 7f19c8729700 1 -- start start 2023-06-23T01:08:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c8729700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac003fe0 0x7f19c01536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c8729700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00c5460 0x7f19c0145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c8729700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c0156330 0x7f19c01567b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c8729700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c0156cf0 con 0x7f19c00c5460 2023-06-23T01:08:55.713 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c8729700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00c6830 con 0x7f19ac003fe0 2023-06-23T01:08:55.714 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c8729700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19c00c5c20 con 0x7f19c0156330 2023-06-23T01:08:55.714 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00c5460 0x7f19c0145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.714 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00c5460 0x7f19c0145f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:56636/0 (socket says 172.21.15.39:56636) 2023-06-23T01:08:55.714 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 -- 172.21.15.39:0/89063602 learned_addr learned my addr 172.21.15.39:0/89063602 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:55.714 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c64c5700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac003fe0 0x7f19c01536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.714 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c6cc6700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c0156330 0x7f19c01567b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.715 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 -- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c0156330 msgr2=0x7f19c01567b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.715 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c0156330 0x7f19c01567b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.715 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 -- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac003fe0 msgr2=0x7f19c01536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:55.716 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac003fe0 0x7f19c01536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:55.716 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 -- 172.21.15.39:0/89063602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19c00c5f00 con 0x7f19c00c5460 2023-06-23T01:08:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c5cc4700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00c5460 0x7f19c0145f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19ac004730 tx=0x7f19b001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19b67fc700 1 -- 172.21.15.39:0/89063602 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0022070 con 0x7f19c00c5460 2023-06-23T01:08:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19c00c61e0 con 0x7f19c00c5460 2023-06-23T01:08:55.717 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19c00c58d0 con 0x7f19c00c5460 2023-06-23T01:08:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19b67fc700 1 -- 172.21.15.39:0/89063602 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19b000eb90 con 0x7f19c00c5460 2023-06-23T01:08:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.700+0000 7f19b67fc700 1 -- 172.21.15.39:0/89063602 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19b0027470 con 0x7f19c00c5460 2023-06-23T01:08:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.701+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19c004c1b0 con 0x7f19c00c5460 2023-06-23T01:08:55.718 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.702+0000 7f19b67fc700 1 -- 172.21.15.39:0/89063602 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f19b0027820 con 0x7f19c00c5460 2023-06-23T01:08:55.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.702+0000 7f19b67fc700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f19a4066e90 0x7f19a4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:55.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.702+0000 7f19b67fc700 1 -- 172.21.15.39:0/89063602 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f19b0085b80 con 0x7f19c00c5460 2023-06-23T01:08:55.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.702+0000 7f19c64c5700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f19a4066e90 0x7f19a4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:55.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.703+0000 7f19c64c5700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f19a4066e90 0x7f19a4069340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19bc006f00 tx=0x7f19bc0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:55.719 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.706+0000 7f19b67fc700 1 -- 172.21.15.39:0/89063602 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f19b0053200 con 0x7f19c00c5460 2023-06-23T01:08:55.902 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:55.898+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f19c015bd70 con 0x7f19c00c5460 2023-06-23T01:08:56.560 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.556+0000 7f19b67fc700 1 -- 172.21.15.39:0/89063602 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f19b002c0a0 con 0x7f19c00c5460 2023-06-23T01:08:56.563 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f19a4066e90 msgr2=0x7f19a4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:56.564 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f19a4066e90 0x7f19a4069340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19bc006f00 tx=0x7f19bc0091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.564 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00c5460 msgr2=0x7f19c0145f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:56.564 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00c5460 0x7f19c0145f30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f19ac004730 tx=0x7f19b001a000 comp rx=0 tx=0).stop 2023-06-23T01:08:56.564 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 shutdown_connections 2023-06-23T01:08:56.564 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f19c0156330 0x7f19c01567b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.565 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f19a4066e90 0x7f19a4069340 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.565 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f19ac003fe0 0x7f19c01536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.565 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 --2- 172.21.15.39:0/89063602 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f19c00c5460 0x7f19c0145f30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.565 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 >> 172.21.15.39:0/89063602 conn(0x7f19c0067450 msgr2=0x7f19c00cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:56.565 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 shutdown_connections 2023-06-23T01:08:56.565 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.559+0000 7f19c8729700 1 -- 172.21.15.39:0/89063602 wait complete. 2023-06-23T01:08:56.566 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'rbd' created 2023-06-23T01:08:56.576 DEBUG:teuthology.orchestra.run.smithi039:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-06-23T01:08:56.699 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.695+0000 7f8f045e7700 1 Processor -- start 2023-06-23T01:08:56.699 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.695+0000 7f8f045e7700 1 -- start start 2023-06-23T01:08:56.701 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f045e7700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8efc0b2510 0x7f8efc15d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:56.701 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f045e7700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc15e030 con 0x7f8efc153ce0 2023-06-23T01:08:56.701 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f045e7700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8efc0cf500 con 0x7f8efc153910 2023-06-23T01:08:56.702 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f045e7700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8efc0cebd0 con 0x7f8efc0b2510 2023-06-23T01:08:56.702 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f02b84700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8efc0b2510 0x7f8efc15d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:56.702 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f01b82700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f8efc153ce0 0x7f8efc0b1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52588/0 (socket says 172.21.15.39:52588) 2023-06-23T01:08:56.702 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f02b84700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f8efc0b2510 0x7f8efc15d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58318/0 (socket says 172.21.15.39:58318) 2023-06-23T01:08:56.702 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f01b82700 1 -- 172.21.15.39:0/3492232064 learned_addr learned my addr 172.21.15.39:0/3492232064 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:56.703 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f02b84700 1 -- 172.21.15.39:0/3492232064 >> v1:172.21.15.73:6789/0 conn(0x7f8efc153910 legacy=0x7f8efc0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:56.703 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f02b84700 1 -- 172.21.15.39:0/3492232064 >> v1:172.21.15.39:6789/0 conn(0x7f8efc153ce0 legacy=0x7f8efc0b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:56.703 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f01381700 1 -- 172.21.15.39:0/3492232064 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939070450 0 0) 0x7f8efc15e030 con 0x7f8efc153ce0 2023-06-23T01:08:56.703 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f02b84700 1 -- 172.21.15.39:0/3492232064 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8efc0ceeb0 con 0x7f8efc0b2510 2023-06-23T01:08:56.703 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f02b84700 1 --2- 172.21.15.39:0/3492232064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0b2510 0x7f8efc15d290 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40098e0 tx=0x7f8ef40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddee093cf5c6cea server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:56.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.696+0000 7f8f01381700 1 -- 172.21.15.39:0/3492232064 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef4009e80 con 0x7f8efc0b2510 2023-06-23T01:08:56.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f01381700 1 -- 172.21.15.39:0/3492232064 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef400b860 con 0x7f8efc0b2510 2023-06-23T01:08:56.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f01381700 1 -- 172.21.15.39:0/3492232064 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef400bbe0 con 0x7f8efc0b2510 2023-06-23T01:08:56.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f045e7700 1 -- 172.21.15.39:0/3492232064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0b2510 msgr2=0x7f8efc15d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:56.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f045e7700 1 --2- 172.21.15.39:0/3492232064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0b2510 0x7f8efc15d290 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8ef40098e0 tx=0x7f8ef40091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.704 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f045e7700 1 -- 172.21.15.39:0/3492232064 shutdown_connections 2023-06-23T01:08:56.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f045e7700 1 --2- 172.21.15.39:0/3492232064 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc0b2510 0x7f8efc15d290 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f045e7700 1 -- 172.21.15.39:0/3492232064 >> 172.21.15.39:0/3492232064 conn(0x7f8efc067450 msgr2=0x7f8efc067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:56.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f045e7700 1 -- 172.21.15.39:0/3492232064 shutdown_connections 2023-06-23T01:08:56.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.697+0000 7f8f045e7700 1 -- 172.21.15.39:0/3492232064 wait complete. 2023-06-23T01:08:56.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.698+0000 7f8f045e7700 1 Processor -- start 2023-06-23T01:08:56.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.698+0000 7f8f045e7700 1 -- start start 2023-06-23T01:08:56.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f045e7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8efc0b2510 0x7f8efc15ae20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:56.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f045e7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc153910 0x7f8efc14d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:56.708 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f045e7700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f8efc153ce0 0x7f8efc0b02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:56.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f045e7700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0b0810 con 0x7f8efc153910 2023-06-23T01:08:56.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f045e7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc0b12c0 con 0x7f8efc0b2510 2023-06-23T01:08:56.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f045e7700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8efc169ab0 con 0x7f8efc153ce0 2023-06-23T01:08:56.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f01b82700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc153910 0x7f8efc14d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:56.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f01b82700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc153910 0x7f8efc14d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58320/0 (socket says 172.21.15.39:58320) 2023-06-23T01:08:56.709 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f01b82700 1 -- 172.21.15.39:0/1933884302 learned_addr learned my addr 172.21.15.39:0/1933884302 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:56.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f02b84700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f8efc153ce0 0x7f8efc0b02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:56.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f02383700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8efc0b2510 0x7f8efc15ae20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:56.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f01b82700 1 -- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f8efc153ce0 msgr2=0x7f8efc0b02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:56.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f01b82700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f8efc153ce0 0x7f8efc0b02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f01b82700 1 -- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8efc0b2510 msgr2=0x7f8efc15ae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:56.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f01b82700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8efc0b2510 0x7f8efc15ae20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:56.710 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.699+0000 7f8f01b82700 1 -- 172.21.15.39:0/1933884302 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ef4009420 con 0x7f8efc153910 2023-06-23T01:08:56.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.700+0000 7f8f01b82700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc153910 0x7f8efc14d040 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000c840 tx=0x7f8ef0007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:56.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.700+0000 7f8efa7fc700 1 -- 172.21.15.39:0/1933884302 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0005c00 con 0x7f8efc153910 2023-06-23T01:08:56.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.700+0000 7f8efa7fc700 1 -- 172.21.15.39:0/1933884302 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ef0005d60 con 0x7f8efc153910 2023-06-23T01:08:56.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.700+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8efc0b0a20 con 0x7f8efc153910 2023-06-23T01:08:56.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.700+0000 7f8efa7fc700 1 -- 172.21.15.39:0/1933884302 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ef0007320 con 0x7f8efc153910 2023-06-23T01:08:56.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.700+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8efc0b0fe0 con 0x7f8efc153910 2023-06-23T01:08:56.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.701+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8efc04c1b0 con 0x7f8efc153910 2023-06-23T01:08:56.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.701+0000 7f8efa7fc700 1 -- 172.21.15.39:0/1933884302 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f8ef001b3b0 con 0x7f8efc153910 2023-06-23T01:08:56.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.702+0000 7f8efa7fc700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8ee8066e90 0x7f8ee8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:56.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.702+0000 7f8efa7fc700 1 -- 172.21.15.39:0/1933884302 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5195+0+0 (secure 0 0 0) 0x7f8ef0083680 con 0x7f8efc153910 2023-06-23T01:08:56.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.702+0000 7f8f02383700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8ee8066e90 0x7f8ee8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:56.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.702+0000 7f8f02383700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8ee8066e90 0x7f8ee8069340 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007fd0 tx=0x7f8eec009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:56.713 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.706+0000 7f8efa7fc700 1 -- 172.21.15.39:0/1933884302 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f8ef0050bb0 con 0x7f8efc153910 2023-06-23T01:08:56.904 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:56.900+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8efc169bf0 con 0x7f8efc153910 2023-06-23T01:08:57.567 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.563+0000 7f8efa7fc700 1 -- 172.21.15.39:0/1933884302 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f8ef00170c0 con 0x7f8efc153910 2023-06-23T01:08:57.571 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8ee8066e90 msgr2=0x7f8ee8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:57.571 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8ee8066e90 0x7f8ee8069340 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8eec007fd0 tx=0x7f8eec009040 comp rx=0 tx=0).stop 2023-06-23T01:08:57.571 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc153910 msgr2=0x7f8efc14d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:57.571 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc153910 0x7f8efc14d040 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8ef000c840 tx=0x7f8ef0007d90 comp rx=0 tx=0).stop 2023-06-23T01:08:57.571 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 shutdown_connections 2023-06-23T01:08:57.571 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f8efc153ce0 0x7f8efc0b02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.572 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8ee8066e90 0x7f8ee8069340 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.572 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8efc0b2510 0x7f8efc15ae20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.572 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 --2- 172.21.15.39:0/1933884302 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f8efc153910 0x7f8efc14d040 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.572 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.566+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 >> 172.21.15.39:0/1933884302 conn(0x7f8efc067450 msgr2=0x7f8efc0b3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:57.572 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.567+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 shutdown_connections 2023-06-23T01:08:57.572 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.567+0000 7f8f045e7700 1 -- 172.21.15.39:0/1933884302 wait complete. 2023-06-23T01:08:57.573 INFO:teuthology.orchestra.run.smithi039.stderr:enabled application 'rbd' on pool 'rbd' 2023-06-23T01:08:57.588 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-06-23T01:08:57.588 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-23T01:08:57.589 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-23T01:08:57.592 INFO:tasks.ceph.mds.a:Started 2023-06-23T01:08:57.592 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-23T01:08:57.593 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-23T01:08:57.595 INFO:tasks.ceph.mds.c:Started 2023-06-23T01:08:57.595 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-23T01:08:57.596 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-23T01:08:57.599 INFO:tasks.ceph.mds.b:Started 2023-06-23T01:08:57.599 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-23T01:08:57.600 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-23T01:08:57.602 INFO:tasks.ceph.mds.d:Started 2023-06-23T01:08:57.603 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-06-23T01:08:57.604 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:08:57.671 INFO:tasks.ceph.mds.a.smithi039.stdout:starting mds.a at 2023-06-23T01:08:57.673 INFO:tasks.ceph.mds.c.smithi039.stdout:starting mds.c at 2023-06-23T01:08:57.680 INFO:tasks.ceph.mds.d.smithi073.stdout:starting mds.d at 2023-06-23T01:08:57.681 INFO:tasks.ceph.mds.b.smithi073.stdout:starting mds.b at 2023-06-23T01:08:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.733+0000 7f093986c700 1 Processor -- start 2023-06-23T01:08:57.737 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.733+0000 7f093986c700 1 -- start start 2023-06-23T01:08:57.739 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f093986c700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f0934153990 0x7f09340cc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:57.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f093986c700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f09340cb6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:57.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f093986c700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09340cd010 con 0x7f0934153d60 2023-06-23T01:08:57.740 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f093986c700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09340cbbe0 con 0x7f0934153990 2023-06-23T01:08:57.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f093986c700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09340cd2f0 con 0x7f09340cb200 2023-06-23T01:08:57.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f09327fc700 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f0934153d60 0x7f093415a4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.39:57210/0 (socket says 172.21.15.39:57210) 2023-06-23T01:08:57.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f09337fe700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f09340cb6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:57.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f09327fc700 1 -- 172.21.15.39:0/2608455424 learned_addr learned my addr 172.21.15.39:0/2608455424 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:57.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f0932ffd700 1 --2- 172.21.15.39:0/2608455424 >> v2:172.21.15.73:3300/0 conn(0x7f0934153990 0x7f09340cc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:57.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.734+0000 7f0931ffb700 1 -- 172.21.15.39:0/2608455424 <== mon.1 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3134697495 0 0) 0x7f09340cd010 con 0x7f0934153d60 2023-06-23T01:08:57.741 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0931ffb700 1 -- 172.21.15.39:0/2608455424 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f091c003570 con 0x7f0934153d60 2023-06-23T01:08:57.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0932ffd700 1 -- 172.21.15.39:0/2608455424 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 msgr2=0x7f09340cb6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:57.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0932ffd700 1 --2- 172.21.15.39:0/2608455424 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f09340cb6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0932ffd700 1 -- 172.21.15.39:0/2608455424 >> v1:172.21.15.73:6789/0 conn(0x7f0934153d60 legacy=0x7f093415a4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:57.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0932ffd700 1 -- 172.21.15.39:0/2608455424 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09340cd5d0 con 0x7f0934153990 2023-06-23T01:08:57.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f09337fe700 1 --2- 172.21.15.39:0/2608455424 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f09340cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:57.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0932ffd700 1 --2- 172.21.15.39:0/2608455424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0934153990 0x7f09340cc960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f09240098d0 tx=0x7f0924009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9182fb10168baed server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:57.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0931ffb700 1 -- 172.21.15.39:0/2608455424 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924005c10 con 0x7f0934153990 2023-06-23T01:08:57.742 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0931ffb700 1 -- 172.21.15.39:0/2608455424 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924005d70 con 0x7f0934153990 2023-06-23T01:08:57.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f0931ffb700 1 -- 172.21.15.39:0/2608455424 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0924006260 con 0x7f0934153990 2023-06-23T01:08:57.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f093986c700 1 -- 172.21.15.39:0/2608455424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0934153990 msgr2=0x7f09340cc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:57.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f093986c700 1 --2- 172.21.15.39:0/2608455424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0934153990 0x7f09340cc960 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f09240098d0 tx=0x7f0924009120 comp rx=0 tx=0).stop 2023-06-23T01:08:57.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f093986c700 1 -- 172.21.15.39:0/2608455424 shutdown_connections 2023-06-23T01:08:57.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f093986c700 1 --2- 172.21.15.39:0/2608455424 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f09340cb6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f093986c700 1 --2- 172.21.15.39:0/2608455424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0934153990 0x7f09340cc960 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.743 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f093986c700 1 -- 172.21.15.39:0/2608455424 >> 172.21.15.39:0/2608455424 conn(0x7f093406a060 msgr2=0x7f093406a430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:57.744 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f093986c700 1 -- 172.21.15.39:0/2608455424 shutdown_connections 2023-06-23T01:08:57.744 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.735+0000 7f093986c700 1 -- 172.21.15.39:0/2608455424 wait complete. 2023-06-23T01:08:57.746 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f093986c700 1 Processor -- start 2023-06-23T01:08:57.746 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f093986c700 1 -- start start 2023-06-23T01:08:57.746 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f093986c700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f0934145c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:57.746 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f093986c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0934153990 0x7f09340b59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:57.746 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f093986c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f093416db50 0x7f09340c5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f093986c700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f093416d6c0 con 0x7f0934153990 2023-06-23T01:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f093986c700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09340c8a00 con 0x7f093416db50 2023-06-23T01:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f093986c700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f093416d070 con 0x7f09340cb200 2023-06-23T01:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.737+0000 7f09327fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0934153990 0x7f09340b59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09327fc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0934153990 0x7f09340b59f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58338/0 (socket says 172.21.15.39:58338) 2023-06-23T01:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09327fc700 1 -- 172.21.15.39:0/3375181504 learned_addr learned my addr 172.21.15.39:0/3375181504 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:57.747 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f0932ffd700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f0934145c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09337fe700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f093416db50 0x7f09340c5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09327fc700 1 -- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 msgr2=0x7f0934145c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09327fc700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f0934145c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09327fc700 1 -- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f093416db50 msgr2=0x7f09340c5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09327fc700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f093416db50 0x7f09340c5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09327fc700 1 -- 172.21.15.39:0/3375181504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092c009a20 con 0x7f0934153990 2023-06-23T01:08:57.748 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f09327fc700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0934153990 0x7f09340b59f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f092800c560 tx=0x7f0928007dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:57.749 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f091affd700 1 -- 172.21.15.39:0/3375181504 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928005c00 con 0x7f0934153990 2023-06-23T01:08:57.749 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09240093c0 con 0x7f0934153990 2023-06-23T01:08:57.749 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f091affd700 1 -- 172.21.15.39:0/3375181504 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0928005d60 con 0x7f0934153990 2023-06-23T01:08:57.749 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f091affd700 1 -- 172.21.15.39:0/3375181504 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0928007300 con 0x7f0934153990 2023-06-23T01:08:57.749 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.738+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09341461c0 con 0x7f0934153990 2023-06-23T01:08:57.749 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.739+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093416ced0 con 0x7f0934153990 2023-06-23T01:08:57.749 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.740+0000 7f091affd700 1 -- 172.21.15.39:0/3375181504 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f09280076b0 con 0x7f0934153990 2023-06-23T01:08:57.749 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.740+0000 7f091affd700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0920066e90 0x7f0920069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:57.750 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.740+0000 7f091affd700 1 -- 172.21.15.39:0/3375181504 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f0928083680 con 0x7f0934153990 2023-06-23T01:08:57.750 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.740+0000 7f0932ffd700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0920066e90 0x7f0920069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:57.750 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.741+0000 7f0932ffd700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0920066e90 0x7f0920069340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0924009160 tx=0x7f0924016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:57.750 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.744+0000 7f091affd700 1 -- 172.21.15.39:0/3375181504 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f093416ced0 con 0x7f0934153990 2023-06-23T01:08:57.941 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.937+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f093416ced0 con 0x7f0934153990 2023-06-23T01:08:57.942 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.938+0000 7f091affd700 1 -- 172.21.15.39:0/3375181504 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+13708 (secure 0 0 0) 0x7f093416ced0 con 0x7f0934153990 2023-06-23T01:08:57.942 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:08:57.942 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":18,"fsid":"314a4b16-8059-4f2c-986c-1624c6de1afb","created":"2023-06-23T01:08:37.443723+0000","modified":"2023-06-23T01:08:57.552571+0000","last_up_change":"2023-06-23T01:08:48.497542+0000","last_in_change":"2023-06-23T01:08:40.002994+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:08:49.048645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:08:55.930040+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a33e557d-f70e-4a1c-a10e-bc7f351dee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":117964},{"type":"v1","addr":"172.21.15.39:6817","nonce":117964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":117964},{"type":"v1","addr":"172.21.15.39:6819","nonce":117964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":117964},{"type":"v1","addr":"172.21.15.39:6823","nonce":117964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":117964},{"type":"v1","addr":"172.21.15.39:6821","nonce":117964}]},"public_addr":"172.21.15.39:6817/117964","cluster_addr":"172.21.15.39:6819/117964","heartbeat_back_addr":"172.21.15.39:6823/117964","heartbeat_front_addr":"172.21.15.39:6821/117964","state":["exists","up"]},{"osd":1,"uuid":"757696fc-3c60-4f7f-822d-a8ff1e89534d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":117963},{"type":"v1","addr":"172.21.15.39:6801","nonce":117963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":117963},{"type":"v1","addr":"172.21.15.39:6803","nonce":117963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":117963},{"type":"v1","addr":"172.21.15.39:6807","nonce":117963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":117963},{"type":"v1","addr":"172.21.15.39:6805","nonce":117963}]},"public_addr":"172.21.15.39:6801/117963","cluster_addr":"172.21.15.39:6803/117963","heartbeat_back_addr":"172.21.15.39:6807/117963","heartbeat_front_addr":"172.21.15.39:6805/117963","state":["exists","up"]},{"osd":2,"uuid":"cffcd6d2-b414-40b0-8967-26421b058b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":117974},{"type":"v1","addr":"172.21.15.39:6825","nonce":117974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":117974},{"type":"v1","addr":"172.21.15.39:6827","nonce":117974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":117974},{"type":"v1","addr":"172.21.15.39:6831","nonce":117974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":117974},{"type":"v1","addr":"172.21.15.39:6829","nonce":117974}]},"public_addr":"172.21.15.39:6825/117974","cluster_addr":"172.21.15.39:6827/117974","heartbeat_back_addr":"172.21.15.39:6831/117974","heartbeat_front_addr":"172.21.15.39:6829/117974","state":["exists","up"]},{"osd":3,"uuid":"5a91897e-5924-4d57-ac5e-bfd06cf4ff93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":117973},{"type":"v1","addr":"172.21.15.39:6809","nonce":117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":117973},{"type":"v1","addr":"172.21.15.39:6811","nonce":117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":117973},{"type":"v1","addr":"172.21.15.39:6815","nonce":117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":117973},{"type":"v1","addr":"172.21.15.39:6813","nonce":117973}]},"public_addr":"172.21.15.39:6809/117973","cluster_addr":"172.21.15.39:6811/117973","heartbeat_back_addr":"172.21.15.39:6815/117973","heartbeat_front_addr":"172.21.15.39:6813/117973","state":["exists","up"]},{"osd":4,"uuid":"67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":118146},{"type":"v1","addr":"172.21.15.73:6801","nonce":118146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":118146},{"type":"v1","addr":"172.21.15.73:6803","nonce":118146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":118146},{"type":"v1","addr":"172.21.15.73:6807","nonce":118146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":118146},{"type":"v1","addr":"172.21.15.73:6805","nonce":118146}]},"public_addr":"172.21.15.73:6801/118146","cluster_addr":"172.21.15.73:6803/118146","heartbeat_back_addr":"172.21.15.73:6807/118146","heartbeat_front_addr":"172.21.15.73:6805/118146","state":["exists","up"]},{"osd":5,"uuid":"390058ef-ef47-475e-af72-8e2f51ee184c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":118148},{"type":"v1","addr":"172.21.15.73:6817","nonce":118148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":118148},{"type":"v1","addr":"172.21.15.73:6819","nonce":118148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":118148},{"type":"v1","addr":"172.21.15.73:6823","nonce":118148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":118148},{"type":"v1","addr":"172.21.15.73:6821","nonce":118148}]},"public_addr":"172.21.15.73:6817/118148","cluster_addr":"172.21.15.73:6819/118148","heartbeat_back_addr":"172.21.15.73:6823/118148","heartbeat_front_addr":"172.21.15.73:6821/118148","state":["exists","up"]},{"osd":6,"uuid":"c34ad563-5018-4cf4-830f-2ac56a3a5e5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":118147},{"type":"v1","addr":"172.21.15.73:6809","nonce":118147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":118147},{"type":"v1","addr":"172.21.15.73:6811","nonce":118147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":118147},{"type":"v1","addr":"172.21.15.73:6815","nonce":118147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":118147},{"type":"v1","addr":"172.21.15.73:6813","nonce":118147}]},"public_addr":"172.21.15.73:6809/118147","cluster_addr":"172.21.15.73:6811/118147","heartbeat_back_addr":"172.21.15.73:6815/118147","heartbeat_front_addr":"172.21.15.73:6813/118147","state":["exists","up"]},{"osd":7,"uuid":"e2023d49-2e44-412c-8cd1-c67447567c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":118189},{"type":"v1","addr":"172.21.15.73:6825","nonce":118189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":118189},{"type":"v1","addr":"172.21.15.73:6827","nonce":118189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":118189},{"type":"v1","addr":"172.21.15.73:6831","nonce":118189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":118189},{"type":"v1","addr":"172.21.15.73:6829","nonce":118189}]},"public_addr":"172.21.15.73:6825/118189","cluster_addr":"172.21.15.73:6827/118189","heartbeat_back_addr":"172.21.15.73:6831/118189","heartbeat_front_addr":"172.21.15.73:6829/118189","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.198991+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.140721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.167693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.101533+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.095800+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.189388+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.133372+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.190656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:08:57.944 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0920066e90 msgr2=0x7f0920069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:57.945 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0920066e90 0x7f0920069340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0924009160 tx=0x7f0924016000 comp rx=0 tx=0).stop 2023-06-23T01:08:57.945 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0934153990 msgr2=0x7f09340b59f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:57.945 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0934153990 0x7f09340b59f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f092800c560 tx=0x7f0928007dd0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.945 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 shutdown_connections 2023-06-23T01:08:57.945 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0920066e90 0x7f0920069340 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.945 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f093416db50 0x7f09340c5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.945 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f09340cb200 0x7f0934145c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.946 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 --2- 172.21.15.39:0/3375181504 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0934153990 0x7f09340b59f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:57.946 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 >> 172.21.15.39:0/3375181504 conn(0x7f093406a060 msgr2=0x7f0934151c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:57.946 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 shutdown_connections 2023-06-23T01:08:57.946 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:57.940+0000 7f093986c700 1 -- 172.21.15.39:0/3375181504 wait complete. 2023-06-23T01:08:57.956 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:08:49.048645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-23T01:08:55.930040+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-06-23T01:08:57.957 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:08:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.073+0000 7fe7bea7b700 1 Processor -- start 2023-06-23T01:08:58.078 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.074+0000 7fe7bea7b700 1 -- start start 2023-06-23T01:08:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.074+0000 7fe7bea7b700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2720 0x7fe7b81579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.074+0000 7fe7bea7b700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe7b815af60 0x7fe7b815d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.074+0000 7fe7bea7b700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7b815c0e0 con 0x7fe7b80b2af0 2023-06-23T01:08:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.074+0000 7fe7bea7b700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7b815b4d0 con 0x7fe7b815af60 2023-06-23T01:08:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.074+0000 7fe7bea7b700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815b7b0 con 0x7fe7b80b2720 2023-06-23T01:08:58.081 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.074+0000 7fe7bd018700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fe7b815af60 0x7fe7b815d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.074+0000 7fe7b7fff700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fe7b80b2af0 0x7fe7b80cd6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52594/0 (socket says 172.21.15.39:52594) 2023-06-23T01:08:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7b7fff700 1 -- 172.21.15.39:0/66039037 learned_addr learned my addr 172.21.15.39:0/66039037 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7bc817700 1 --2- 172.21.15.39:0/66039037 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2720 0x7fe7b81579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7bd018700 1 -- 172.21.15.39:0/66039037 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2720 msgr2=0x7fe7b81579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7bd018700 1 --2- 172.21.15.39:0/66039037 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2720 0x7fe7b81579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7bd018700 1 -- 172.21.15.39:0/66039037 >> v1:172.21.15.39:6789/0 conn(0x7fe7b80b2af0 legacy=0x7fe7b80cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.082 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7b77fe700 1 -- 172.21.15.39:0/66039037 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846347813 0 0) 0x7fe7b815c0e0 con 0x7fe7b80b2af0 2023-06-23T01:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7bd018700 1 -- 172.21.15.39:0/66039037 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b815ba90 con 0x7fe7b815af60 2023-06-23T01:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7bd018700 1 --2- 172.21.15.39:0/66039037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b815af60 0x7fe7b815d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009e60 tx=0x7fe7b0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46c9362e6ce4c426 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7b77fe700 1 -- 172.21.15.39:0/66039037 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b000e070 con 0x7fe7b815af60 2023-06-23T01:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7b77fe700 1 -- 172.21.15.39:0/66039037 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7b000bb60 con 0x7fe7b815af60 2023-06-23T01:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.075+0000 7fe7b77fe700 1 -- 172.21.15.39:0/66039037 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7b0012430 con 0x7fe7b815af60 2023-06-23T01:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.076+0000 7fe7bea7b700 1 -- 172.21.15.39:0/66039037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b815af60 msgr2=0x7fe7b815d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.076+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/66039037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b815af60 0x7fe7b815d390 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe7b0009e60 tx=0x7fe7b0009780 comp rx=0 tx=0).stop 2023-06-23T01:08:58.083 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.076+0000 7fe7bea7b700 1 -- 172.21.15.39:0/66039037 shutdown_connections 2023-06-23T01:08:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.076+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/66039037 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2720 0x7fe7b81579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.076+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/66039037 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b815af60 0x7fe7b815d390 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.076+0000 7fe7bea7b700 1 -- 172.21.15.39:0/66039037 >> 172.21.15.39:0/66039037 conn(0x7fe7b8067450 msgr2=0x7fe7b8067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.076+0000 7fe7bea7b700 1 -- 172.21.15.39:0/66039037 shutdown_connections 2023-06-23T01:08:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.076+0000 7fe7bea7b700 1 -- 172.21.15.39:0/66039037 wait complete. 2023-06-23T01:08:58.084 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.077+0000 7fe7bea7b700 1 Processor -- start 2023-06-23T01:08:58.085 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:08:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.077+0000 7fe7bea7b700 1 -- start start 2023-06-23T01:08:58.087 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bea7b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b80b2720 0x7fe7b81551c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bea7b700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2af0 0x7fe7b80b5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bea7b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7b815af60 0x7fe7b81582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bea7b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815f450 con 0x7fe7b80b2720 2023-06-23T01:08:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bea7b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815e840 con 0x7fe7b815af60 2023-06-23T01:08:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bea7b700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7b815eb20 con 0x7fe7b80b2af0 2023-06-23T01:08:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b80b2720 0x7fe7b81551c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.088 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b80b2720 0x7fe7b81551c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58358/0 (socket says 172.21.15.39:58358) 2023-06-23T01:08:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 -- 172.21.15.39:0/3240434487 learned_addr learned my addr 172.21.15.39:0/3240434487 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7b7fff700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2af0 0x7fe7b80b5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bd018700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7b815af60 0x7fe7b81582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 -- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2af0 msgr2=0x7fe7b80b5830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2af0 0x7fe7b80b5830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 -- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7b815af60 msgr2=0x7fe7b81582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.089 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7b815af60 0x7fe7b81582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 -- 172.21.15.39:0/3240434487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7b0009a20 con 0x7fe7b80b2720 2023-06-23T01:08:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bd018700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7b815af60 0x7fe7b81582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:08:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bc817700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b80b2720 0x7fe7b81551c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800ec80 tx=0x7fe7a80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7b4ff9700 1 -- 172.21.15.39:0/3240434487 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a800c850 con 0x7fe7b80b2720 2023-06-23T01:08:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7b4ff9700 1 -- 172.21.15.39:0/3240434487 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7a800c9b0 con 0x7fe7b80b2720 2023-06-23T01:08:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7b815ee00 con 0x7fe7b80b2720 2023-06-23T01:08:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7b4ff9700 1 -- 172.21.15.39:0/3240434487 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7a800cd50 con 0x7fe7b80b2720 2023-06-23T01:08:58.090 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.078+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7b80c74f0 con 0x7fe7b80b2720 2023-06-23T01:08:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.079+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7b804c1b0 con 0x7fe7b80b2720 2023-06-23T01:08:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.080+0000 7fe7b4ff9700 1 -- 172.21.15.39:0/3240434487 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fe7a8013680 con 0x7fe7b80b2720 2023-06-23T01:08:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.080+0000 7fe7b4ff9700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fe7a4066e90 0x7fe7a4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.080+0000 7fe7b4ff9700 1 -- 172.21.15.39:0/3240434487 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fe7a8084530 con 0x7fe7b80b2720 2023-06-23T01:08:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.080+0000 7fe7b7fff700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fe7a4066e90 0x7fe7a4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.091 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.081+0000 7fe7b7fff700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fe7a4066e90 0x7fe7a4069340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe7b816cde0 tx=0x7fe7ac007d30 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.092 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.085+0000 7fe7b4ff9700 1 -- 172.21.15.39:0/3240434487 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fe7a8051a50 con 0x7fe7b80b2720 2023-06-23T01:08:58.093 DEBUG:teuthology.orchestra.run.smithi073:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:08:58.282 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.278+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe7b815f2a0 con 0x7fe7b80b2720 2023-06-23T01:08:58.283 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.279+0000 7fe7b4ff9700 1 -- 172.21.15.39:0/3240434487 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe7a8055070 con 0x7fe7b80b2720 2023-06-23T01:08:58.283 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2023-06-23T01:08:58.285 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fe7a4066e90 msgr2=0x7fe7a4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.285 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fe7a4066e90 0x7fe7a4069340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe7b816cde0 tx=0x7fe7ac007d30 comp rx=0 tx=0).stop 2023-06-23T01:08:58.285 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b80b2720 msgr2=0x7fe7b81551c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.285 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b80b2720 0x7fe7b81551c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fe7a800ec80 tx=0x7fe7a80091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.285 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 shutdown_connections 2023-06-23T01:08:58.285 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fe7a4066e90 0x7fe7a4069340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.286 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7b815af60 0x7fe7b81582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.286 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fe7b80b2720 0x7fe7b81551c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.286 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 --2- 172.21.15.39:0/3240434487 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fe7b80b2af0 0x7fe7b80b5830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.286 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.280+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 >> 172.21.15.39:0/3240434487 conn(0x7fe7b8067450 msgr2=0x7fe7b8156e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:58.286 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.281+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 shutdown_connections 2023-06-23T01:08:58.286 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.281+0000 7fe7bea7b700 1 -- 172.21.15.39:0/3240434487 wait complete. 2023-06-23T01:08:58.297 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T01:08:58.417 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.413+0000 7fd03942d700 1 Processor -- start 2023-06-23T01:08:58.418 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.413+0000 7fd03942d700 1 -- start start 2023-06-23T01:08:58.419 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd03942d700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 0x7fd03415a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.420 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd03942d700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd034067b60 0x7fd03415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.420 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd03942d700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0340ce670 con 0x7fd034157070 2023-06-23T01:08:58.420 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd03942d700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd03415de90 con 0x7fd034067b60 2023-06-23T01:08:58.420 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd03942d700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd03415d3e0 con 0x7fd034156ca0 2023-06-23T01:08:58.420 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd038c2c700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd034067b60 0x7fd03415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.420 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd038c2c700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd034067b60 0x7fd03415cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58362/0 (socket says 172.21.15.39:58362) 2023-06-23T01:08:58.420 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd038c2c700 1 -- 172.21.15.39:0/3283182531 learned_addr learned my addr 172.21.15.39:0/3283182531 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:58.421 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd033fff700 1 --2- 172.21.15.39:0/3283182531 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 0x7fd03415a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.421 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd038c2c700 1 -- 172.21.15.39:0/3283182531 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 msgr2=0x7fd03415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:58.421 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd038c2c700 1 --2- 172.21.15.39:0/3283182531 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 0x7fd03415a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.421 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.414+0000 7fd038c2c700 1 -- 172.21.15.39:0/3283182531 >> v1:172.21.15.73:6789/0 conn(0x7fd034157070 legacy=0x7fd034067450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.421 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd038c2c700 1 -- 172.21.15.39:0/3283182531 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd03415d5f0 con 0x7fd034067b60 2023-06-23T01:08:58.421 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd038c2c700 1 --2- 172.21.15.39:0/3283182531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034067b60 0x7fd03415cea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009f20 tx=0x7fd02c009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b946de56d259960 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.421 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd032ffd700 1 -- 172.21.15.39:0/3283182531 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c00e070 con 0x7fd034067b60 2023-06-23T01:08:58.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd032ffd700 1 -- 172.21.15.39:0/3283182531 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c00b9d0 con 0x7fd034067b60 2023-06-23T01:08:58.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd032ffd700 1 -- 172.21.15.39:0/3283182531 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c0092e0 con 0x7fd034067b60 2023-06-23T01:08:58.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd03942d700 1 -- 172.21.15.39:0/3283182531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034067b60 msgr2=0x7fd03415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd03942d700 1 --2- 172.21.15.39:0/3283182531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034067b60 0x7fd03415cea0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd02c009f20 tx=0x7fd02c009840 comp rx=0 tx=0).stop 2023-06-23T01:08:58.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd03942d700 1 -- 172.21.15.39:0/3283182531 shutdown_connections 2023-06-23T01:08:58.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd03942d700 1 --2- 172.21.15.39:0/3283182531 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034067b60 0x7fd03415cea0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd03942d700 1 --2- 172.21.15.39:0/3283182531 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 0x7fd03415a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.422 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd03942d700 1 -- 172.21.15.39:0/3283182531 >> 172.21.15.39:0/3283182531 conn(0x7fd0340d0240 msgr2=0x7fd0340d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:58.423 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd03942d700 1 -- 172.21.15.39:0/3283182531 shutdown_connections 2023-06-23T01:08:58.423 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.415+0000 7fd03942d700 1 -- 172.21.15.39:0/3283182531 wait complete. 2023-06-23T01:08:58.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.416+0000 7fd03942d700 1 Processor -- start 2023-06-23T01:08:58.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd03942d700 1 -- start start 2023-06-23T01:08:58.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd03942d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd034067b60 0x7fd034155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.425 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd03942d700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 0x7fd0340b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd03942d700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034157070 0x7fd0340b52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd03942d700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0340aff00 con 0x7fd034157070 2023-06-23T01:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd03942d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0340b0150 con 0x7fd034067b60 2023-06-23T01:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd03942d700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0340b0430 con 0x7fd034156ca0 2023-06-23T01:08:58.426 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd038c2c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034157070 0x7fd0340b52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd038c2c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034157070 0x7fd0340b52c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58376/0 (socket says 172.21.15.39:58376) 2023-06-23T01:08:58.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd038c2c700 1 -- 172.21.15.39:0/1348035057 learned_addr learned my addr 172.21.15.39:0/1348035057 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:58.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.417+0000 7fd033fff700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd034067b60 0x7fd034155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.427 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd0337fe700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 0x7fd0340b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd038c2c700 1 -- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 msgr2=0x7fd0340b4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd038c2c700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 0x7fd0340b4d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd038c2c700 1 -- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd034067b60 msgr2=0x7fd034155b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd038c2c700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd034067b60 0x7fd034155b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd038c2c700 1 -- 172.21.15.39:0/1348035057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02c009ae0 con 0x7fd034157070 2023-06-23T01:08:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd038c2c700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034157070 0x7fd0340b52c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd03406b080 tx=0x7fd02c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.428 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd02bfff700 1 -- 172.21.15.39:0/1348035057 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c008ad0 con 0x7fd034157070 2023-06-23T01:08:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd02bfff700 1 -- 172.21.15.39:0/1348035057 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c008c30 con 0x7fd034157070 2023-06-23T01:08:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0340b0710 con 0x7fd034157070 2023-06-23T01:08:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd02bfff700 1 -- 172.21.15.39:0/1348035057 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c01c410 con 0x7fd034157070 2023-06-23T01:08:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.418+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0340b0d50 con 0x7fd034157070 2023-06-23T01:08:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.419+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03404c1b0 con 0x7fd034157070 2023-06-23T01:08:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.419+0000 7fd02bfff700 1 -- 172.21.15.39:0/1348035057 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fd02c003040 con 0x7fd034157070 2023-06-23T01:08:58.429 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.420+0000 7fd02bfff700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd014066e90 0x7fd014069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.430 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.420+0000 7fd02bfff700 1 -- 172.21.15.39:0/1348035057 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd02c085d40 con 0x7fd034157070 2023-06-23T01:08:58.430 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.420+0000 7fd033fff700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd014066e90 0x7fd014069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.430 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.421+0000 7fd033fff700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd014066e90 0x7fd014069340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd01c009db0 tx=0x7fd01c0091a0 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.430 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.425+0000 7fd02bfff700 1 -- 172.21.15.39:0/1348035057 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fd02c053260 con 0x7fd034157070 2023-06-23T01:08:58.625 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.621+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd0340b0ee0 con 0x7fd034157070 2023-06-23T01:08:58.625 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.621+0000 7fd02bfff700 1 -- 172.21.15.39:0/1348035057 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd02c056880 con 0x7fd034157070 2023-06-23T01:08:58.625 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 8 2023-06-23T01:08:58.626 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd014066e90 msgr2=0x7fd014069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd014066e90 0x7fd014069340 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fd01c009db0 tx=0x7fd01c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034157070 msgr2=0x7fd0340b52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034157070 0x7fd0340b52c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd03406b080 tx=0x7fd02c018000 comp rx=0 tx=0).stop 2023-06-23T01:08:58.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 shutdown_connections 2023-06-23T01:08:58.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd034157070 0x7fd0340b52c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.627 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd014066e90 0x7fd014069340 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd034067b60 0x7fd034155b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 --2- 172.21.15.39:0/1348035057 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd034156ca0 0x7fd0340b4d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 >> 172.21.15.39:0/1348035057 conn(0x7fd0340d0240 msgr2=0x7fd0340cf7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:58.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 shutdown_connections 2023-06-23T01:08:58.628 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.623+0000 7fd03942d700 1 -- 172.21.15.39:0/1348035057 wait complete. 2023-06-23T01:08:58.638 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:08:58.757 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.754+0000 7f2e0f161700 1 Processor -- start 2023-06-23T01:08:58.758 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.754+0000 7f2e0f161700 1 -- start start 2023-06-23T01:08:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.755+0000 7f2e0f161700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e080b2720 0x7f2e081579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.755+0000 7f2e0f161700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0815c0e0 con 0x7f2e080b2af0 2023-06-23T01:08:58.759 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.755+0000 7f2e0f161700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0815b4d0 con 0x7f2e0815af60 2023-06-23T01:08:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.755+0000 7f2e0f161700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0815b7b0 con 0x7f2e080b2720 2023-06-23T01:08:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.755+0000 7f2e07fff700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f2e080b2af0 0x7f2e080cd6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52604/0 (socket says 172.21.15.39:52604) 2023-06-23T01:08:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.755+0000 7f2e07fff700 1 -- 172.21.15.39:0/942626150 learned_addr learned my addr 172.21.15.39:0/942626150 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.755+0000 7f2e0cefd700 1 --2- 172.21.15.39:0/942626150 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e080b2720 0x7f2e081579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e077fe700 1 -- 172.21.15.39:0/942626150 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4140488525 0 0) 0x7f2e0815c0e0 con 0x7f2e080b2af0 2023-06-23T01:08:58.760 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e077fe700 1 -- 172.21.15.39:0/942626150 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df4003570 con 0x7f2e080b2af0 2023-06-23T01:08:58.761 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e077fe700 1 -- 172.21.15.39:0/942626150 <== mon.1 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659879364 0 0) 0x7f2e0815b4d0 con 0x7f2e0815af60 2023-06-23T01:08:58.761 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e077fe700 1 -- 172.21.15.39:0/942626150 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e0815c0e0 con 0x7f2e0815af60 2023-06-23T01:08:58.761 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e0cefd700 1 -- 172.21.15.39:0/942626150 >> v1:172.21.15.73:6789/0 conn(0x7f2e0815af60 legacy=0x7f2e0815d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.761 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e0cefd700 1 -- 172.21.15.39:0/942626150 >> v1:172.21.15.39:6789/0 conn(0x7f2e080b2af0 legacy=0x7f2e080cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.761 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e0cefd700 1 -- 172.21.15.39:0/942626150 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0815ba90 con 0x7f2e080b2720 2023-06-23T01:08:58.761 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e0cefd700 1 --2- 172.21.15.39:0/942626150 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e080b2720 0x7f2e081579d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009910 tx=0x7f2df00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d9cd396af9d89ce server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.762 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e077fe700 1 -- 172.21.15.39:0/942626150 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0011630 con 0x7f2e080b2720 2023-06-23T01:08:58.762 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e077fe700 1 -- 172.21.15.39:0/942626150 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df0011790 con 0x7f2e080b2720 2023-06-23T01:08:58.762 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.756+0000 7f2e077fe700 1 -- 172.21.15.39:0/942626150 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0011b10 con 0x7f2e080b2720 2023-06-23T01:08:58.764 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.757+0000 7f2e0f161700 1 -- 172.21.15.39:0/942626150 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e080b2720 msgr2=0x7f2e081579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.764 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.757+0000 7f2e0f161700 1 --2- 172.21.15.39:0/942626150 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e080b2720 0x7f2e081579d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2df0009910 tx=0x7f2df00091a0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.764 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.757+0000 7f2e0f161700 1 -- 172.21.15.39:0/942626150 shutdown_connections 2023-06-23T01:08:58.764 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.757+0000 7f2e0f161700 1 --2- 172.21.15.39:0/942626150 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e080b2720 0x7f2e081579d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.765 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.757+0000 7f2e0f161700 1 -- 172.21.15.39:0/942626150 >> 172.21.15.39:0/942626150 conn(0x7f2e08067450 msgr2=0x7f2e08067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:58.765 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.757+0000 7f2e0f161700 1 -- 172.21.15.39:0/942626150 shutdown_connections 2023-06-23T01:08:58.765 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.757+0000 7f2e0f161700 1 -- 172.21.15.39:0/942626150 wait complete. 2023-06-23T01:08:58.765 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.758+0000 7f2e0f161700 1 Processor -- start 2023-06-23T01:08:58.765 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.758+0000 7f2e0f161700 1 -- start start 2023-06-23T01:08:58.765 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0f161700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b2720 0x7f2e080d07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.765 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0f161700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2e080b2af0 0x7f2e080ce7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0f161700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e0815af60 0x7f2e080cf360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0f161700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e08169cd0 con 0x7f2e080b2720 2023-06-23T01:08:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0f161700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0816d760 con 0x7f2e080b2af0 2023-06-23T01:08:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0f161700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0816da40 con 0x7f2e0815af60 2023-06-23T01:08:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b2720 0x7f2e080d07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b2720 0x7f2e080d07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58384/0 (socket says 172.21.15.39:58384) 2023-06-23T01:08:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 -- 172.21.15.39:0/433231667 learned_addr learned my addr 172.21.15.39:0/433231667 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:58.766 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e07fff700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2e080b2af0 0x7f2e080ce7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0d6fe700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e0815af60 0x7f2e080cf360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 -- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e0815af60 msgr2=0x7f2e080cf360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e0815af60 0x7f2e080cf360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 -- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2e080b2af0 msgr2=0x7f2e080ce7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2e080b2af0 0x7f2e080ce7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 -- 172.21.15.39:0/433231667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2df0009420 con 0x7f2e080b2720 2023-06-23T01:08:58.767 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e0cefd700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b2720 0x7f2e080d07c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e08065560 tx=0x7f2df0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.759+0000 7f2e04ff9700 1 -- 172.21.15.39:0/433231667 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0011900 con 0x7f2e080b2720 2023-06-23T01:08:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.760+0000 7f2e04ff9700 1 -- 172.21.15.39:0/433231667 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2df0011a60 con 0x7f2e080b2720 2023-06-23T01:08:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.760+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e0816dd20 con 0x7f2e080b2720 2023-06-23T01:08:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.760+0000 7f2e04ff9700 1 -- 172.21.15.39:0/433231667 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2df0025640 con 0x7f2e080b2720 2023-06-23T01:08:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.760+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e080d0d00 con 0x7f2e080b2720 2023-06-23T01:08:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.761+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e080b20c0 con 0x7f2e080b2720 2023-06-23T01:08:58.768 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.761+0000 7f2e04ff9700 1 -- 172.21.15.39:0/433231667 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f2df0025a30 con 0x7f2e080b2720 2023-06-23T01:08:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.761+0000 7f2e04ff9700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f2de8066e90 0x7f2de8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.762+0000 7f2e04ff9700 1 -- 172.21.15.39:0/433231667 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2df00859a0 con 0x7f2e080b2720 2023-06-23T01:08:58.769 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.762+0000 7f2e07fff700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f2de8066e90 0x7f2de8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.762+0000 7f2e07fff700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f2de8066e90 0x7f2de8069340 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2e081551d0 tx=0x7f2df8009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:58.770 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:58.766+0000 7f2e04ff9700 1 -- 172.21.15.39:0/433231667 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f2df0052ec0 con 0x7f2e080b2720 2023-06-23T01:08:59.003 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.000+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e08169ed0 con 0x7f2e080b2720 2023-06-23T01:08:59.004 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.001+0000 7f2e04ff9700 1 -- 172.21.15.39:0/433231667 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3524 (secure 0 0 0) 0x7f2df0015de0 con 0x7f2e080b2720 2023-06-23T01:08:59.004 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:08:59.004 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4232,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6833/1143164070","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6832","nonce":1143164070},{"type":"v1","addr":"172.21.15.39:6833","nonce":1143164070}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/2338262264","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":2338262264},{"type":"v1","addr":"172.21.15.73:6835","nonce":2338262264}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4284,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.39:6835/2607627279","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6834","nonce":2607627279},{"type":"v1","addr":"172.21.15.39:6835","nonce":2607627279}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4315,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/1280936597","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":1280936597},{"type":"v1","addr":"172.21.15.73:6837","nonce":1280936597}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-23T01:08:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f2de8066e90 msgr2=0x7f2de8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f2de8066e90 0x7f2de8069340 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2e081551d0 tx=0x7f2df8009040 comp rx=0 tx=0).stop 2023-06-23T01:08:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b2720 msgr2=0x7f2e080d07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b2720 0x7f2e080d07c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2e08065560 tx=0x7f2df0017000 comp rx=0 tx=0).stop 2023-06-23T01:08:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 shutdown_connections 2023-06-23T01:08:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f2e0815af60 0x7f2e080cf360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.006 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f2e080b2720 0x7f2e080d07c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f2de8066e90 0x7f2de8069340 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 --2- 172.21.15.39:0/433231667 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2e080b2af0 0x7f2e080ce7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.002+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 >> 172.21.15.39:0/433231667 conn(0x7f2e08067450 msgr2=0x7f2e08159ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.003+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 shutdown_connections 2023-06-23T01:08:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.003+0000 7f2e0f161700 1 -- 172.21.15.39:0/433231667 wait complete. 2023-06-23T01:08:59.007 INFO:teuthology.orchestra.run.smithi039.stderr:dumped fsmap epoch 3 2023-06-23T01:08:59.018 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:08:59.136 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.133+0000 7f957adb8700 1 Processor -- start 2023-06-23T01:08:59.136 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.133+0000 7f957adb8700 1 -- start start 2023-06-23T01:08:59.138 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f957adb8700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f957415ca40 0x7f95740d0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.138 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f957adb8700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f957415ce10 0x7f95740cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f957adb8700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f95740cf210 0x7f957415d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f957adb8700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95740cc930 con 0x7f957415ca40 2023-06-23T01:08:59.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f957adb8700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95740d03e0 con 0x7f95740cf210 2023-06-23T01:08:59.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f957adb8700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95740cf7d0 con 0x7f957415ce10 2023-06-23T01:08:59.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f957415ca40 0x7f95740d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f957415ca40 0x7f95740d0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58390/0 (socket says 172.21.15.39:58390) 2023-06-23T01:08:59.139 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 -- 172.21.15.39:0/3659566681 learned_addr learned my addr 172.21.15.39:0/3659566681 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:59.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9579355700 1 --2- 172.21.15.39:0/3659566681 >> v2:172.21.15.73:3300/0 conn(0x7f95740cf210 0x7f957415d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9573fff700 1 --2- 172.21.15.39:0/3659566681 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f957415ce10 0x7f95740cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 -- 172.21.15.39:0/3659566681 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f957415ce10 msgr2=0x7f95740cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:59.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 --2- 172.21.15.39:0/3659566681 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f957415ce10 0x7f95740cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 -- 172.21.15.39:0/3659566681 >> v2:172.21.15.73:3300/0 conn(0x7f95740cf210 msgr2=0x7f957415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:59.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 --2- 172.21.15.39:0/3659566681 >> v2:172.21.15.73:3300/0 conn(0x7f95740cf210 0x7f957415d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 -- 172.21.15.39:0/3659566681 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95740cfab0 con 0x7f957415ca40 2023-06-23T01:08:59.140 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9573fff700 1 --2- 172.21.15.39:0/3659566681 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f957415ce10 0x7f95740cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:08:59.141 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9579355700 1 --2- 172.21.15.39:0/3659566681 >> v2:172.21.15.73:3300/0 conn(0x7f95740cf210 0x7f957415d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:08:59.141 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f9578b54700 1 --2- 172.21.15.39:0/3659566681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ca40 0x7f95740d0870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f95740d0350 tx=0x7f9568009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80405146fc30e0f9 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.141 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f95737fe700 1 -- 172.21.15.39:0/3659566681 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9568005c10 con 0x7f957415ca40 2023-06-23T01:08:59.141 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f95737fe700 1 -- 172.21.15.39:0/3659566681 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9568005d70 con 0x7f957415ca40 2023-06-23T01:08:59.141 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.134+0000 7f95737fe700 1 -- 172.21.15.39:0/3659566681 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956800b880 con 0x7f957415ca40 2023-06-23T01:08:59.141 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 -- 172.21.15.39:0/3659566681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ca40 msgr2=0x7f95740d0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.142 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 --2- 172.21.15.39:0/3659566681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ca40 0x7f95740d0870 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f95740d0350 tx=0x7f9568009220 comp rx=0 tx=0).stop 2023-06-23T01:08:59.142 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 -- 172.21.15.39:0/3659566681 shutdown_connections 2023-06-23T01:08:59.142 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 --2- 172.21.15.39:0/3659566681 >> v2:172.21.15.73:3300/0 conn(0x7f95740cf210 0x7f957415d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.142 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 --2- 172.21.15.39:0/3659566681 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ca40 0x7f95740d0870 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.142 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 --2- 172.21.15.39:0/3659566681 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f957415ce10 0x7f95740cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.142 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 -- 172.21.15.39:0/3659566681 >> 172.21.15.39:0/3659566681 conn(0x7f9574067450 msgr2=0x7f9574067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:59.142 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 -- 172.21.15.39:0/3659566681 shutdown_connections 2023-06-23T01:08:59.142 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.135+0000 7f957adb8700 1 -- 172.21.15.39:0/3659566681 wait complete. 2023-06-23T01:08:59.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.136+0000 7f957adb8700 1 Processor -- start 2023-06-23T01:08:59.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.136+0000 7f957adb8700 1 -- start start 2023-06-23T01:08:59.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f957adb8700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f95740cf210 0x7f95740c9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f957adb8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957415ca40 0x7f957414bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f957adb8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ce10 0x7f95740b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.145 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f957adb8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9574150340 con 0x7f957415ce10 2023-06-23T01:08:59.146 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f957adb8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95740ca420 con 0x7f957415ca40 2023-06-23T01:08:59.146 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f957adb8700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95740b3ed0 con 0x7f95740cf210 2023-06-23T01:08:59.146 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9579355700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ce10 0x7f95740b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.146 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9579355700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ce10 0x7f95740b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58398/0 (socket says 172.21.15.39:58398) 2023-06-23T01:08:59.146 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9579355700 1 -- 172.21.15.39:0/2499004030 learned_addr learned my addr 172.21.15.39:0/2499004030 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:59.146 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9578b54700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f95740cf210 0x7f95740c9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.146 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9573fff700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957415ca40 0x7f957414bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.146 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9579355700 1 -- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f95740cf210 msgr2=0x7f95740c9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9579355700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f95740cf210 0x7f95740c9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9579355700 1 -- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957415ca40 msgr2=0x7f957414bcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9579355700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957415ca40 0x7f957414bcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.137+0000 7f9579355700 1 -- 172.21.15.39:0/2499004030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9568009480 con 0x7f957415ce10 2023-06-23T01:08:59.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.138+0000 7f9579355700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ce10 0x7f95740b1870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f956c00d180 tx=0x7f956c009a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.138+0000 7f9570ff9700 1 -- 172.21.15.39:0/2499004030 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c00dcc0 con 0x7f957415ce10 2023-06-23T01:08:59.147 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.138+0000 7f9570ff9700 1 -- 172.21.15.39:0/2499004030 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f956c00f040 con 0x7f957415ce10 2023-06-23T01:08:59.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.138+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95740b41b0 con 0x7f957415ce10 2023-06-23T01:08:59.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.138+0000 7f9570ff9700 1 -- 172.21.15.39:0/2499004030 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f956c015690 con 0x7f957415ce10 2023-06-23T01:08:59.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.138+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95740b3ba0 con 0x7f957415ce10 2023-06-23T01:08:59.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.139+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957404c1b0 con 0x7f957415ce10 2023-06-23T01:08:59.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.139+0000 7f9570ff9700 1 -- 172.21.15.39:0/2499004030 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f956c01e3b0 con 0x7f957415ce10 2023-06-23T01:08:59.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.140+0000 7f9570ff9700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f9558066e90 0x7f9558069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.140+0000 7f9570ff9700 1 -- 172.21.15.39:0/2499004030 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f956c0847a0 con 0x7f957415ce10 2023-06-23T01:08:59.148 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.140+0000 7f9578b54700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f9558066e90 0x7f9558069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.140+0000 7f9578b54700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f9558066e90 0x7f9558069340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9568009260 tx=0x7f9568016000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.149 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.144+0000 7f9570ff9700 1 -- 172.21.15.39:0/2499004030 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f956c051c20 con 0x7f957415ce10 2023-06-23T01:08:59.345 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.342+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f95740b46d0 con 0x7f957415ce10 2023-06-23T01:08:59.346 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.343+0000 7f9570ff9700 1 -- 172.21.15.39:0/2499004030 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+13695 (secure 0 0 0) 0x7f95740b46d0 con 0x7f957415ce10 2023-06-23T01:08:59.346 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:08:59.346 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":19,"fsid":"314a4b16-8059-4f2c-986c-1624c6de1afb","created":"2023-06-23T01:08:37.443723+0000","modified":"2023-06-23T01:08:58.559179+0000","last_up_change":"2023-06-23T01:08:48.497542+0000","last_in_change":"2023-06-23T01:08:40.002994+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:08:49.048645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:08:55.930040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"a33e557d-f70e-4a1c-a10e-bc7f351dee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":117964},{"type":"v1","addr":"172.21.15.39:6817","nonce":117964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":117964},{"type":"v1","addr":"172.21.15.39:6819","nonce":117964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":117964},{"type":"v1","addr":"172.21.15.39:6823","nonce":117964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":117964},{"type":"v1","addr":"172.21.15.39:6821","nonce":117964}]},"public_addr":"172.21.15.39:6817/117964","cluster_addr":"172.21.15.39:6819/117964","heartbeat_back_addr":"172.21.15.39:6823/117964","heartbeat_front_addr":"172.21.15.39:6821/117964","state":["exists","up"]},{"osd":1,"uuid":"757696fc-3c60-4f7f-822d-a8ff1e89534d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":117963},{"type":"v1","addr":"172.21.15.39:6801","nonce":117963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":117963},{"type":"v1","addr":"172.21.15.39:6803","nonce":117963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":117963},{"type":"v1","addr":"172.21.15.39:6807","nonce":117963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":117963},{"type":"v1","addr":"172.21.15.39:6805","nonce":117963}]},"public_addr":"172.21.15.39:6801/117963","cluster_addr":"172.21.15.39:6803/117963","heartbeat_back_addr":"172.21.15.39:6807/117963","heartbeat_front_addr":"172.21.15.39:6805/117963","state":["exists","up"]},{"osd":2,"uuid":"cffcd6d2-b414-40b0-8967-26421b058b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":117974},{"type":"v1","addr":"172.21.15.39:6825","nonce":117974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":117974},{"type":"v1","addr":"172.21.15.39:6827","nonce":117974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":117974},{"type":"v1","addr":"172.21.15.39:6831","nonce":117974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":117974},{"type":"v1","addr":"172.21.15.39:6829","nonce":117974}]},"public_addr":"172.21.15.39:6825/117974","cluster_addr":"172.21.15.39:6827/117974","heartbeat_back_addr":"172.21.15.39:6831/117974","heartbeat_front_addr":"172.21.15.39:6829/117974","state":["exists","up"]},{"osd":3,"uuid":"5a91897e-5924-4d57-ac5e-bfd06cf4ff93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":117973},{"type":"v1","addr":"172.21.15.39:6809","nonce":117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":117973},{"type":"v1","addr":"172.21.15.39:6811","nonce":117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":117973},{"type":"v1","addr":"172.21.15.39:6815","nonce":117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":117973},{"type":"v1","addr":"172.21.15.39:6813","nonce":117973}]},"public_addr":"172.21.15.39:6809/117973","cluster_addr":"172.21.15.39:6811/117973","heartbeat_back_addr":"172.21.15.39:6815/117973","heartbeat_front_addr":"172.21.15.39:6813/117973","state":["exists","up"]},{"osd":4,"uuid":"67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":118146},{"type":"v1","addr":"172.21.15.73:6801","nonce":118146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":118146},{"type":"v1","addr":"172.21.15.73:6803","nonce":118146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":118146},{"type":"v1","addr":"172.21.15.73:6807","nonce":118146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":118146},{"type":"v1","addr":"172.21.15.73:6805","nonce":118146}]},"public_addr":"172.21.15.73:6801/118146","cluster_addr":"172.21.15.73:6803/118146","heartbeat_back_addr":"172.21.15.73:6807/118146","heartbeat_front_addr":"172.21.15.73:6805/118146","state":["exists","up"]},{"osd":5,"uuid":"390058ef-ef47-475e-af72-8e2f51ee184c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":118148},{"type":"v1","addr":"172.21.15.73:6817","nonce":118148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":118148},{"type":"v1","addr":"172.21.15.73:6819","nonce":118148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":118148},{"type":"v1","addr":"172.21.15.73:6823","nonce":118148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":118148},{"type":"v1","addr":"172.21.15.73:6821","nonce":118148}]},"public_addr":"172.21.15.73:6817/118148","cluster_addr":"172.21.15.73:6819/118148","heartbeat_back_addr":"172.21.15.73:6823/118148","heartbeat_front_addr":"172.21.15.73:6821/118148","state":["exists","up"]},{"osd":6,"uuid":"c34ad563-5018-4cf4-830f-2ac56a3a5e5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":118147},{"type":"v1","addr":"172.21.15.73:6809","nonce":118147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":118147},{"type":"v1","addr":"172.21.15.73:6811","nonce":118147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":118147},{"type":"v1","addr":"172.21.15.73:6815","nonce":118147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":118147},{"type":"v1","addr":"172.21.15.73:6813","nonce":118147}]},"public_addr":"172.21.15.73:6809/118147","cluster_addr":"172.21.15.73:6811/118147","heartbeat_back_addr":"172.21.15.73:6815/118147","heartbeat_front_addr":"172.21.15.73:6813/118147","state":["exists","up"]},{"osd":7,"uuid":"e2023d49-2e44-412c-8cd1-c67447567c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":118189},{"type":"v1","addr":"172.21.15.73:6825","nonce":118189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":118189},{"type":"v1","addr":"172.21.15.73:6827","nonce":118189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":118189},{"type":"v1","addr":"172.21.15.73:6831","nonce":118189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":118189},{"type":"v1","addr":"172.21.15.73:6829","nonce":118189}]},"public_addr":"172.21.15.73:6825/118189","cluster_addr":"172.21.15.73:6827/118189","heartbeat_back_addr":"172.21.15.73:6831/118189","heartbeat_front_addr":"172.21.15.73:6829/118189","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.198991+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.140721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.167693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.101533+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.095800+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.189388+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.133372+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.190656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:08:59.349 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f9558066e90 msgr2=0x7f9558069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.349 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f9558066e90 0x7f9558069340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f9568009260 tx=0x7f9568016000 comp rx=0 tx=0).stop 2023-06-23T01:08:59.349 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ce10 msgr2=0x7f95740b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.349 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ce10 0x7f95740b1870 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f956c00d180 tx=0x7f956c009a50 comp rx=0 tx=0).stop 2023-06-23T01:08:59.349 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 shutdown_connections 2023-06-23T01:08:59.349 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f957415ce10 0x7f95740b1870 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.349 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f95740cf210 0x7f95740c9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f9558066e90 0x7f9558069340 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 --2- 172.21.15.39:0/2499004030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957415ca40 0x7f957414bcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 >> 172.21.15.39:0/2499004030 conn(0x7f9574067450 msgr2=0x7f957414cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:59.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 shutdown_connections 2023-06-23T01:08:59.350 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.345+0000 7f957adb8700 1 -- 172.21.15.39:0/2499004030 wait complete. 2023-06-23T01:08:59.360 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:08:49.048645+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-23T01:08:55.930040+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-06-23T01:08:59.361 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:08:59.482 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.479+0000 7faf14ffc700 1 Processor -- start 2023-06-23T01:08:59.483 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.479+0000 7faf14ffc700 1 -- start start 2023-06-23T01:08:59.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.480+0000 7faf14ffc700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 0x7faf100cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.480+0000 7faf14ffc700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faf1015af60 0x7faf1015d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.480+0000 7faf14ffc700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf1015c0e0 con 0x7faf100b2720 2023-06-23T01:08:59.484 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.480+0000 7faf14ffc700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf1015b4d0 con 0x7faf1015af60 2023-06-23T01:08:59.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.480+0000 7faf14ffc700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1015b7b0 con 0x7faf100b2af0 2023-06-23T01:08:59.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0ed9d700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faf1015af60 0x7faf1015d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0ed9d700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7faf1015af60 0x7faf1015d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58408/0 (socket says 172.21.15.39:58408) 2023-06-23T01:08:59.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0ed9d700 1 -- 172.21.15.39:0/1715261278 learned_addr learned my addr 172.21.15.39:0/1715261278 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:59.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0dd9b700 1 --2- 172.21.15.39:0/1715261278 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 0x7faf100cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0ed9d700 1 -- 172.21.15.39:0/1715261278 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 msgr2=0x7faf100cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:59.485 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0ed9d700 1 --2- 172.21.15.39:0/1715261278 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 0x7faf100cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0ed9d700 1 -- 172.21.15.39:0/1715261278 >> v1:172.21.15.73:6789/0 conn(0x7faf100b2720 legacy=0x7faf101579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0ed9d700 1 -- 172.21.15.39:0/1715261278 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf1015ba90 con 0x7faf1015af60 2023-06-23T01:08:59.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0ed9d700 1 --2- 172.21.15.39:0/1715261278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf1015af60 0x7faf1015d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf080099b0 tx=0x7faf08009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac3564bf1a0958b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0d59a700 1 -- 172.21.15.39:0/1715261278 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0800b860 con 0x7faf1015af60 2023-06-23T01:08:59.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0d59a700 1 -- 172.21.15.39:0/1715261278 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0800b9c0 con 0x7faf1015af60 2023-06-23T01:08:59.486 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.481+0000 7faf0d59a700 1 -- 172.21.15.39:0/1715261278 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0800bd60 con 0x7faf1015af60 2023-06-23T01:08:59.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.482+0000 7faf14ffc700 1 -- 172.21.15.39:0/1715261278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf1015af60 msgr2=0x7faf1015d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.488 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.482+0000 7faf14ffc700 1 --2- 172.21.15.39:0/1715261278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf1015af60 0x7faf1015d390 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7faf080099b0 tx=0x7faf08009250 comp rx=0 tx=0).stop 2023-06-23T01:08:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.482+0000 7faf14ffc700 1 -- 172.21.15.39:0/1715261278 shutdown_connections 2023-06-23T01:08:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.482+0000 7faf14ffc700 1 --2- 172.21.15.39:0/1715261278 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf1015af60 0x7faf1015d390 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.482+0000 7faf14ffc700 1 --2- 172.21.15.39:0/1715261278 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 0x7faf100cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.482+0000 7faf14ffc700 1 -- 172.21.15.39:0/1715261278 >> 172.21.15.39:0/1715261278 conn(0x7faf10067450 msgr2=0x7faf10067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.482+0000 7faf14ffc700 1 -- 172.21.15.39:0/1715261278 shutdown_connections 2023-06-23T01:08:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.482+0000 7faf14ffc700 1 -- 172.21.15.39:0/1715261278 wait complete. 2023-06-23T01:08:59.489 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.483+0000 7faf14ffc700 1 Processor -- start 2023-06-23T01:08:59.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.483+0000 7faf14ffc700 1 -- start start 2023-06-23T01:08:59.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.483+0000 7faf14ffc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf100b2720 0x7faf10155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf14ffc700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 0x7faf10145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf14ffc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf1015af60 0x7faf100cf300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf14ffc700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf10156df0 con 0x7faf100b2720 2023-06-23T01:08:59.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf14ffc700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf101561e0 con 0x7faf1015af60 2023-06-23T01:08:59.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf14ffc700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf101564c0 con 0x7faf100b2af0 2023-06-23T01:08:59.490 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf100b2720 0x7faf10155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf100b2720 0x7faf10155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58410/0 (socket says 172.21.15.39:58410) 2023-06-23T01:08:59.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 -- 172.21.15.39:0/167981604 learned_addr learned my addr 172.21.15.39:0/167981604 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:59.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0dd9b700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 0x7faf10145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0ed9d700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf1015af60 0x7faf100cf300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 -- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 msgr2=0x7faf10145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 0x7faf10145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.491 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 -- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf1015af60 msgr2=0x7faf100cf300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf1015af60 0x7faf100cf300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 -- 172.21.15.39:0/167981604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf080094f0 con 0x7faf100b2720 2023-06-23T01:08:59.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf0e59c700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf100b2720 0x7faf10155990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf0000cb10 tx=0x7faf000017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf067fc700 1 -- 172.21.15.39:0/167981604 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0000ce20 con 0x7faf100b2720 2023-06-23T01:08:59.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf067fc700 1 -- 172.21.15.39:0/167981604 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf0000a870 con 0x7faf100b2720 2023-06-23T01:08:59.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf067fc700 1 -- 172.21.15.39:0/167981604 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf0000ac30 con 0x7faf100b2720 2023-06-23T01:08:59.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf101567a0 con 0x7faf100b2720 2023-06-23T01:08:59.492 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.484+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf10155ed0 con 0x7faf100b2720 2023-06-23T01:08:59.493 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.485+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1004c1b0 con 0x7faf100b2720 2023-06-23T01:08:59.493 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.486+0000 7faf067fc700 1 -- 172.21.15.39:0/167981604 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7faf00003030 con 0x7faf100b2720 2023-06-23T01:08:59.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.486+0000 7faf067fc700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7faef0066e90 0x7faef0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.486+0000 7faf067fc700 1 -- 172.21.15.39:0/167981604 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7faf00084390 con 0x7faf100b2720 2023-06-23T01:08:59.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.486+0000 7faf0dd9b700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7faef0066e90 0x7faef0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.487+0000 7faf0dd9b700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7faef0066e90 0x7faef0069340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf101540c0 tx=0x7faef8008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.494 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.490+0000 7faf067fc700 1 -- 172.21.15.39:0/167981604 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7faf00051810 con 0x7faf100b2720 2023-06-23T01:08:59.687 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.684+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7faf1015bd70 con 0x7faf100b2720 2023-06-23T01:08:59.688 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.684+0000 7faf067fc700 1 -- 172.21.15.39:0/167981604 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7faf00054e30 con 0x7faf100b2720 2023-06-23T01:08:59.688 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 1 2023-06-23T01:08:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7faef0066e90 msgr2=0x7faef0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7faef0066e90 0x7faef0069340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7faf101540c0 tx=0x7faef8008040 comp rx=0 tx=0).stop 2023-06-23T01:08:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf100b2720 msgr2=0x7faf10155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.690 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf100b2720 0x7faf10155990 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf0000cb10 tx=0x7faf000017b0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 shutdown_connections 2023-06-23T01:08:59.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7faef0066e90 0x7faef0069340 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf1015af60 0x7faf100cf300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.691 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7faf100b2720 0x7faf10155990 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.686+0000 7faf14ffc700 1 --2- 172.21.15.39:0/167981604 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7faf100b2af0 0x7faf10145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.687+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 >> 172.21.15.39:0/167981604 conn(0x7faf10067450 msgr2=0x7faf100b15c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:59.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.687+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 shutdown_connections 2023-06-23T01:08:59.692 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.687+0000 7faf14ffc700 1 -- 172.21.15.39:0/167981604 wait complete. 2023-06-23T01:08:59.701 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T01:08:59.823 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.820+0000 7f58a359e700 1 Processor -- start 2023-06-23T01:08:59.824 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.820+0000 7f58a359e700 1 -- start start 2023-06-23T01:08:59.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a359e700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f58a4156ca0 0x7f58a415a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a359e700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a359e700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58a40ce670 con 0x7f58a4157070 2023-06-23T01:08:59.826 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a359e700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58a415de90 con 0x7f58a4156ca0 2023-06-23T01:08:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a359e700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a415d3e0 con 0x7f58a4067b60 2023-06-23T01:08:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a1d9b700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f58a4157070 0x7f58a4067450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52618/0 (socket says 172.21.15.39:52618) 2023-06-23T01:08:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a1d9b700 1 -- 172.21.15.39:0/4209320798 learned_addr learned my addr 172.21.15.39:0/4209320798 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:08:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a259c700 1 --2- 172.21.15.39:0/4209320798 >> v2:172.21.15.73:3300/0 conn(0x7f58a4156ca0 0x7f58a415a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a2d9d700 1 --2- 172.21.15.39:0/4209320798 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a159a700 1 -- 172.21.15.39:0/4209320798 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3606163717 0 0) 0x7f58a40ce670 con 0x7f58a4157070 2023-06-23T01:08:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a159a700 1 -- 172.21.15.39:0/4209320798 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f588c003570 con 0x7f58a4157070 2023-06-23T01:08:59.827 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a2d9d700 1 -- 172.21.15.39:0/4209320798 >> v2:172.21.15.73:3300/0 conn(0x7f58a4156ca0 msgr2=0x7f58a415a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:08:59.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a2d9d700 1 --2- 172.21.15.39:0/4209320798 >> v2:172.21.15.73:3300/0 conn(0x7f58a4156ca0 0x7f58a415a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a2d9d700 1 -- 172.21.15.39:0/4209320798 >> v1:172.21.15.39:6789/0 conn(0x7f58a4157070 legacy=0x7f58a4067450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a2d9d700 1 -- 172.21.15.39:0/4209320798 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58a415d5f0 con 0x7f58a4067b60 2023-06-23T01:08:59.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.821+0000 7f58a259c700 1 --2- 172.21.15.39:0/4209320798 >> v2:172.21.15.73:3300/0 conn(0x7f58a4156ca0 0x7f58a415a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:08:59.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a2d9d700 1 --2- 172.21.15.39:0/4209320798 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a415cea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f589c009e60 tx=0x7f589c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2ff190a078d6a45f server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a159a700 1 -- 172.21.15.39:0/4209320798 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00e070 con 0x7f58a4067b60 2023-06-23T01:08:59.828 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a159a700 1 -- 172.21.15.39:0/4209320798 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f589c00bab0 con 0x7f58a4067b60 2023-06-23T01:08:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a159a700 1 -- 172.21.15.39:0/4209320798 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f589c00be30 con 0x7f58a4067b60 2023-06-23T01:08:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a359e700 1 -- 172.21.15.39:0/4209320798 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 msgr2=0x7f58a415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a359e700 1 --2- 172.21.15.39:0/4209320798 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a415cea0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f589c009e60 tx=0x7f589c009780 comp rx=0 tx=0).stop 2023-06-23T01:08:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a359e700 1 -- 172.21.15.39:0/4209320798 shutdown_connections 2023-06-23T01:08:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a359e700 1 --2- 172.21.15.39:0/4209320798 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a415cea0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.829 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a359e700 1 --2- 172.21.15.39:0/4209320798 >> v2:172.21.15.73:3300/0 conn(0x7f58a4156ca0 0x7f58a415a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.830 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a359e700 1 -- 172.21.15.39:0/4209320798 >> 172.21.15.39:0/4209320798 conn(0x7f58a40d0240 msgr2=0x7f58a40d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:08:59.830 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a359e700 1 -- 172.21.15.39:0/4209320798 shutdown_connections 2023-06-23T01:08:59.830 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.822+0000 7f58a359e700 1 -- 172.21.15.39:0/4209320798 wait complete. 2023-06-23T01:08:59.837 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a359e700 1 Processor -- start 2023-06-23T01:08:59.837 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a359e700 1 -- start start 2023-06-23T01:08:59.837 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a359e700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a4144c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.837 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a359e700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58a4156ca0 0x7f58a40c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.837 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a359e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58a4157070 0x7f58a40cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.838 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a359e700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4159540 con 0x7f58a4156ca0 2023-06-23T01:08:59.838 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a359e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58a40b06f0 con 0x7f58a4157070 2023-06-23T01:08:59.838 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a359e700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58a4154360 con 0x7f58a4067b60 2023-06-23T01:08:59.838 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a1d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58a4156ca0 0x7f58a40c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.838 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a1d9b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58a4156ca0 0x7f58a40c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58412/0 (socket says 172.21.15.39:58412) 2023-06-23T01:08:59.838 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.824+0000 7f58a1d9b700 1 -- 172.21.15.39:0/1611046095 learned_addr learned my addr 172.21.15.39:0/1611046095 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:08:59.838 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a2d9d700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58a4157070 0x7f58a40cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.839 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a259c700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a4144c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.839 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a1d9b700 1 -- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 msgr2=0x7f58a4144c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.839 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a1d9b700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a4144c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.839 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a1d9b700 1 -- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58a4157070 msgr2=0x7f58a40cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:08:59.839 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a1d9b700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58a4157070 0x7f58a40cce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:08:59.839 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a1d9b700 1 -- 172.21.15.39:0/1611046095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5894009420 con 0x7f58a4156ca0 2023-06-23T01:08:59.839 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a1d9b700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58a4156ca0 0x7f58a40c9b70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f589800c7f0 tx=0x7f5898007d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.839 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58927fc700 1 -- 172.21.15.39:0/1611046095 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5898005c00 con 0x7f58a4156ca0 2023-06-23T01:08:59.840 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58927fc700 1 -- 172.21.15.39:0/1611046095 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5898005d60 con 0x7f58a4156ca0 2023-06-23T01:08:59.840 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f589c009a20 con 0x7f58a4156ca0 2023-06-23T01:08:59.840 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58927fc700 1 -- 172.21.15.39:0/1611046095 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58980072e0 con 0x7f58a4156ca0 2023-06-23T01:08:59.840 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.825+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58a40b00a0 con 0x7f58a4156ca0 2023-06-23T01:08:59.840 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.826+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a40b08a0 con 0x7f58a4156ca0 2023-06-23T01:08:59.840 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.826+0000 7f58927fc700 1 -- 172.21.15.39:0/1611046095 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f5898007690 con 0x7f58a4156ca0 2023-06-23T01:08:59.840 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.827+0000 7f58927fc700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f5884066e90 0x7f5884069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:08:59.841 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.827+0000 7f58927fc700 1 -- 172.21.15.39:0/1611046095 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f5898083640 con 0x7f58a4156ca0 2023-06-23T01:08:59.841 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.827+0000 7f58a259c700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f5884066e90 0x7f5884069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:08:59.841 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.827+0000 7f58a259c700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f5884066e90 0x7f5884069340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5894006f00 tx=0x7f5894008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:08:59.841 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:08:59.831+0000 7f58927fc700 1 -- 172.21.15.39:0/1611046095 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f589804b590 con 0x7f58a4156ca0 2023-06-23T01:09:00.027 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.024+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f58a404c1b0 con 0x7f58a4156ca0 2023-06-23T01:09:00.028 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.024+0000 7f58927fc700 1 -- 172.21.15.39:0/1611046095 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5898017020 con 0x7f58a4156ca0 2023-06-23T01:09:00.028 INFO:teuthology.orchestra.run.smithi039.stdout:pg_num: 8 2023-06-23T01:09:00.030 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.026+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f5884066e90 msgr2=0x7f5884069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.030 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.026+0000 7f58a359e700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f5884066e90 0x7f5884069340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f5894006f00 tx=0x7f5894008040 comp rx=0 tx=0).stop 2023-06-23T01:09:00.030 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.026+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58a4156ca0 msgr2=0x7f58a40c9b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.030 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.026+0000 7f58a359e700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58a4156ca0 0x7f58a40c9b70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f589800c7f0 tx=0x7f5898007d90 comp rx=0 tx=0).stop 2023-06-23T01:09:00.030 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.026+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 shutdown_connections 2023-06-23T01:09:00.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.027+0000 7f58a359e700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f5884066e90 0x7f5884069340 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.027+0000 7f58a359e700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58a4157070 0x7f58a40cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.027+0000 7f58a359e700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f58a4067b60 0x7f58a4144c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.027+0000 7f58a359e700 1 --2- 172.21.15.39:0/1611046095 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f58a4156ca0 0x7f58a40c9b70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.027+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 >> 172.21.15.39:0/1611046095 conn(0x7f58a40d0240 msgr2=0x7f58a4067450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:00.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.027+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 shutdown_connections 2023-06-23T01:09:00.031 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.027+0000 7f58a359e700 1 -- 172.21.15.39:0/1611046095 wait complete. 2023-06-23T01:09:00.042 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-23T01:09:00.160 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.157+0000 7fa9de520700 1 Processor -- start 2023-06-23T01:09:00.161 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.157+0000 7fa9de520700 1 -- start start 2023-06-23T01:09:00.162 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9de520700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d815a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9de520700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d80ce670 con 0x7fa9d8157070 2023-06-23T01:09:00.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9de520700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9d815d5f0 con 0x7fa9d8156ca0 2023-06-23T01:09:00.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9de520700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d815de90 con 0x7fa9d8067b60 2023-06-23T01:09:00.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9ddd1f700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fa9d8157070 0x7fa9d80ceb00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52622/0 (socket says 172.21.15.39:52622) 2023-06-23T01:09:00.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9ddd1f700 1 -- 172.21.15.39:0/3045588268 learned_addr learned my addr 172.21.15.39:0/3045588268 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:09:00.163 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9dcd1d700 1 --2- 172.21.15.39:0/3045588268 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d815a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9d7fff700 1 -- 172.21.15.39:0/3045588268 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 576163399 0 0) 0x7fa9d80ce670 con 0x7fa9d8157070 2023-06-23T01:09:00.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9d7fff700 1 -- 172.21.15.39:0/3045588268 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c0003570 con 0x7fa9d8157070 2023-06-23T01:09:00.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.158+0000 7fa9d7fff700 1 -- 172.21.15.39:0/3045588268 <== mon.1 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322531641 0 0) 0x7fa9d815d5f0 con 0x7fa9d8156ca0 2023-06-23T01:09:00.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9d7fff700 1 -- 172.21.15.39:0/3045588268 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d80ce670 con 0x7fa9d8156ca0 2023-06-23T01:09:00.164 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9dcd1d700 1 -- 172.21.15.39:0/3045588268 >> v1:172.21.15.73:6789/0 conn(0x7fa9d8156ca0 legacy=0x7fa9d8067450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9dcd1d700 1 -- 172.21.15.39:0/3045588268 >> v1:172.21.15.39:6789/0 conn(0x7fa9d8157070 legacy=0x7fa9d80ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9dcd1d700 1 -- 172.21.15.39:0/3045588268 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9d815d8d0 con 0x7fa9d8067b60 2023-06-23T01:09:00.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9dcd1d700 1 --2- 172.21.15.39:0/3045588268 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d815a3d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009910 tx=0x7fa9cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bcba5ae3ea89b4b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:00.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9d7fff700 1 -- 172.21.15.39:0/3045588268 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc011630 con 0x7fa9d8067b60 2023-06-23T01:09:00.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9d7fff700 1 -- 172.21.15.39:0/3045588268 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9cc011790 con 0x7fa9d8067b60 2023-06-23T01:09:00.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9d7fff700 1 -- 172.21.15.39:0/3045588268 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc011b30 con 0x7fa9d8067b60 2023-06-23T01:09:00.165 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9de520700 1 -- 172.21.15.39:0/3045588268 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 msgr2=0x7fa9d815a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9de520700 1 --2- 172.21.15.39:0/3045588268 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d815a3d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009910 tx=0x7fa9cc0091a0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9de520700 1 -- 172.21.15.39:0/3045588268 shutdown_connections 2023-06-23T01:09:00.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.159+0000 7fa9de520700 1 --2- 172.21.15.39:0/3045588268 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d815a3d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.160+0000 7fa9de520700 1 -- 172.21.15.39:0/3045588268 >> 172.21.15.39:0/3045588268 conn(0x7fa9d80d0240 msgr2=0x7fa9d80d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:00.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.160+0000 7fa9de520700 1 -- 172.21.15.39:0/3045588268 shutdown_connections 2023-06-23T01:09:00.166 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.160+0000 7fa9de520700 1 -- 172.21.15.39:0/3045588268 wait complete. 2023-06-23T01:09:00.168 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.161+0000 7fa9de520700 1 Processor -- start 2023-06-23T01:09:00.168 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.161+0000 7fa9de520700 1 -- start start 2023-06-23T01:09:00.169 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9de520700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d80b39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.169 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9de520700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9d8156ca0 0x7fa9d80b4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.169 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9de520700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d8157070 0x7fa9d8144cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.169 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9de520700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80b3ee0 con 0x7fa9d8157070 2023-06-23T01:09:00.169 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9de520700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80b5060 con 0x7fa9d8156ca0 2023-06-23T01:09:00.169 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9de520700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9d80b4120 con 0x7fa9d8067b60 2023-06-23T01:09:00.169 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d8157070 0x7fa9d8144cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.170 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d8157070 0x7fa9d8144cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58414/0 (socket says 172.21.15.39:58414) 2023-06-23T01:09:00.170 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 -- 172.21.15.39:0/3545208009 learned_addr learned my addr 172.21.15.39:0/3545208009 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:00.170 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9dcd1d700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9d8156ca0 0x7fa9d80b4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.170 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9dd51e700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d80b39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.170 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 -- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 msgr2=0x7fa9d80b39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.170 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d80b39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.170 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 -- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9d8156ca0 msgr2=0x7fa9d80b4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.170 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9d8156ca0 0x7fa9d80b4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.171 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 -- 172.21.15.39:0/3545208009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9cc009420 con 0x7fa9d8157070 2023-06-23T01:09:00.171 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9ddd1f700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d8157070 0x7fa9d8144cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800cb70 tx=0x7fa9c80019e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:00.171 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9d57fa700 1 -- 172.21.15.39:0/3545208009 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800a780 con 0x7fa9d8157070 2023-06-23T01:09:00.171 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9d57fa700 1 -- 172.21.15.39:0/3545208009 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9c800a900 con 0x7fa9d8157070 2023-06-23T01:09:00.171 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9d8157f20 con 0x7fa9d8157070 2023-06-23T01:09:00.171 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9d57fa700 1 -- 172.21.15.39:0/3545208009 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9c800acc0 con 0x7fa9d8157070 2023-06-23T01:09:00.171 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.162+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9d81575f0 con 0x7fa9d8157070 2023-06-23T01:09:00.172 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.163+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9d81580b0 con 0x7fa9d8157070 2023-06-23T01:09:00.172 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.164+0000 7fa9d57fa700 1 -- 172.21.15.39:0/3545208009 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fa9c8007300 con 0x7fa9d8157070 2023-06-23T01:09:00.172 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.164+0000 7fa9d57fa700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fa9c4066e90 0x7fa9c4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.172 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.164+0000 7fa9d57fa700 1 -- 172.21.15.39:0/3545208009 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fa9c80844a0 con 0x7fa9d8157070 2023-06-23T01:09:00.172 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.164+0000 7fa9dd51e700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fa9c4066e90 0x7fa9c4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.172 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.165+0000 7fa9dd51e700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fa9c4066e90 0x7fa9c4069340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8156330 tx=0x7fa9d0007d50 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:00.173 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.168+0000 7fa9d57fa700 1 -- 172.21.15.39:0/3545208009 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fa9c80554d0 con 0x7fa9d8157070 2023-06-23T01:09:00.367 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.364+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa9d804c1b0 con 0x7fa9d8157070 2023-06-23T01:09:00.367 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.364+0000 7fa9d57fa700 1 -- 172.21.15.39:0/3545208009 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7fa9c8055060 con 0x7fa9d8157070 2023-06-23T01:09:00.368 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:09:00.368 INFO:teuthology.orchestra.run.smithi039.stdout:[ 2023-06-23T01:09:00.368 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-06-23T01:09:00.368 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 1, 2023-06-23T01:09:00.368 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": ".mgr" 2023-06-23T01:09:00.369 INFO:teuthology.orchestra.run.smithi039.stdout: }, 2023-06-23T01:09:00.369 INFO:teuthology.orchestra.run.smithi039.stdout: { 2023-06-23T01:09:00.369 INFO:teuthology.orchestra.run.smithi039.stdout: "poolnum": 2, 2023-06-23T01:09:00.369 INFO:teuthology.orchestra.run.smithi039.stdout: "poolname": "rbd" 2023-06-23T01:09:00.369 INFO:teuthology.orchestra.run.smithi039.stdout: } 2023-06-23T01:09:00.369 INFO:teuthology.orchestra.run.smithi039.stdout:] 2023-06-23T01:09:00.370 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fa9c4066e90 msgr2=0x7fa9c4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.370 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fa9c4066e90 0x7fa9c4069340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8156330 tx=0x7fa9d0007d50 comp rx=0 tx=0).stop 2023-06-23T01:09:00.370 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d8157070 msgr2=0x7fa9d8144cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.371 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d8157070 0x7fa9d8144cb0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa9c800cb70 tx=0x7fa9c80019e0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.371 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 shutdown_connections 2023-06-23T01:09:00.371 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fa9d8157070 0x7fa9d8144cb0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.371 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fa9d8067b60 0x7fa9d80b39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.371 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fa9c4066e90 0x7fa9c4069340 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.371 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 --2- 172.21.15.39:0/3545208009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9d8156ca0 0x7fa9d80b4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.371 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 >> 172.21.15.39:0/3545208009 conn(0x7fa9d80d0240 msgr2=0x7fa9d8159910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:00.371 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 shutdown_connections 2023-06-23T01:09:00.372 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.366+0000 7fa9de520700 1 -- 172.21.15.39:0/3545208009 wait complete. 2023-06-23T01:09:00.381 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-23T01:09:00.382 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-23T01:09:00.504 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.501+0000 7f8392f21700 1 Processor -- start 2023-06-23T01:09:00.505 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.501+0000 7f8392f21700 1 -- start start 2023-06-23T01:09:00.506 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.502+0000 7f8392f21700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153910 0x7f838c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.506 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.502+0000 7f8392f21700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f838c0b2510 0x7f838c15d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.507 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.502+0000 7f8392f21700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838c15e030 con 0x7f838c153ce0 2023-06-23T01:09:00.507 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.502+0000 7f8392f21700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c0cf500 con 0x7f838c0b2510 2023-06-23T01:09:00.507 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.502+0000 7f8392f21700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0cfed0 con 0x7f838c153910 2023-06-23T01:09:00.507 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f8390cbd700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153910 0x7f838c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.507 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f83914be700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f838c0b2510 0x7f838c15d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.507 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f838bfff700 1 --1- >> v1:172.21.15.73:6789/0 conn(0x7f838c153ce0 0x7f838c0b1e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.73:6789/0 says I am v1:172.21.15.39:57258/0 (socket says 172.21.15.39:57258) 2023-06-23T01:09:00.508 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f838bfff700 1 -- 172.21.15.39:0/3308708198 learned_addr learned my addr 172.21.15.39:0/3308708198 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:09:00.508 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f838b7fe700 1 -- 172.21.15.39:0/3308708198 <== mon.1 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493783853 0 0) 0x7f838c15e030 con 0x7f838c153ce0 2023-06-23T01:09:00.508 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f838b7fe700 1 -- 172.21.15.39:0/3308708198 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8374003570 con 0x7f838c153ce0 2023-06-23T01:09:00.508 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f83914be700 1 -- 172.21.15.39:0/3308708198 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153910 msgr2=0x7f838c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:00.508 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f83914be700 1 --2- 172.21.15.39:0/3308708198 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153910 0x7f838c0cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.508 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f83914be700 1 -- 172.21.15.39:0/3308708198 >> v1:172.21.15.73:6789/0 conn(0x7f838c153ce0 legacy=0x7f838c0b1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.508 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f83914be700 1 -- 172.21.15.39:0/3308708198 --> v2:172.21.15.73:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c0cebd0 con 0x7f838c0b2510 2023-06-23T01:09:00.509 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f8390cbd700 1 --2- 172.21.15.39:0/3308708198 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153910 0x7f838c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:09:00.509 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f83914be700 1 --2- 172.21.15.39:0/3308708198 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c0b2510 0x7f838c15d290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f837c0098e0 tx=0x7f837c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bd5e7e6a180ac64 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:00.509 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f838b7fe700 1 -- 172.21.15.39:0/3308708198 <== mon.1 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c011630 con 0x7f838c0b2510 2023-06-23T01:09:00.509 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f838b7fe700 1 -- 172.21.15.39:0/3308708198 <== mon.1 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c011790 con 0x7f838c0b2510 2023-06-23T01:09:00.509 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.503+0000 7f838b7fe700 1 -- 172.21.15.39:0/3308708198 <== mon.1 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c011b30 con 0x7f838c0b2510 2023-06-23T01:09:00.511 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.504+0000 7f8392f21700 1 -- 172.21.15.39:0/3308708198 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c0b2510 msgr2=0x7f838c15d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.512 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.504+0000 7f8392f21700 1 --2- 172.21.15.39:0/3308708198 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c0b2510 0x7f838c15d290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f837c0098e0 tx=0x7f837c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.512 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.504+0000 7f8392f21700 1 -- 172.21.15.39:0/3308708198 shutdown_connections 2023-06-23T01:09:00.512 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.504+0000 7f8392f21700 1 --2- 172.21.15.39:0/3308708198 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153910 0x7f838c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.512 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.504+0000 7f8392f21700 1 --2- 172.21.15.39:0/3308708198 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c0b2510 0x7f838c15d290 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.512 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.504+0000 7f8392f21700 1 -- 172.21.15.39:0/3308708198 >> 172.21.15.39:0/3308708198 conn(0x7f838c067450 msgr2=0x7f838c067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:00.512 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.504+0000 7f8392f21700 1 -- 172.21.15.39:0/3308708198 shutdown_connections 2023-06-23T01:09:00.512 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.504+0000 7f8392f21700 1 -- 172.21.15.39:0/3308708198 wait complete. 2023-06-23T01:09:00.513 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.505+0000 7f8392f21700 1 Processor -- start 2023-06-23T01:09:00.513 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.505+0000 7f8392f21700 1 -- start start 2023-06-23T01:09:00.513 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8392f21700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f838c0b2510 0x7f838c14c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.513 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8392f21700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c153910 0x7f838c154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.513 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8392f21700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153ce0 0x7f838c15b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.513 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8392f21700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c156a30 con 0x7f838c0b2510 2023-06-23T01:09:00.513 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8392f21700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c15bd70 con 0x7f838c153910 2023-06-23T01:09:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8392f21700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c156700 con 0x7f838c153ce0 2023-06-23T01:09:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8390cbd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f838c0b2510 0x7f838c14c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8390cbd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f838c0b2510 0x7f838c14c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58416/0 (socket says 172.21.15.39:58416) 2023-06-23T01:09:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8390cbd700 1 -- 172.21.15.39:0/3575684745 learned_addr learned my addr 172.21.15.39:0/3575684745 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f83914be700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153ce0 0x7f838c15b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f838bfff700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c153910 0x7f838c154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.514 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8390cbd700 1 -- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153ce0 msgr2=0x7f838c15b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8390cbd700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153ce0 0x7f838c15b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8390cbd700 1 -- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c153910 msgr2=0x7f838c154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8390cbd700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c153910 0x7f838c154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f8390cbd700 1 -- 172.21.15.39:0/3575684745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8384009a20 con 0x7f838c0b2510 2023-06-23T01:09:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f83914be700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153ce0 0x7f838c15b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:09:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.506+0000 7f838bfff700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c153910 0x7f838c154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:09:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.507+0000 7f8390cbd700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f838c0b2510 0x7f838c14c2b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f838400c1a0 tx=0x7f838400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:00.515 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.507+0000 7f8388ff9700 1 -- 172.21.15.39:0/3575684745 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f838400cce0 con 0x7f838c0b2510 2023-06-23T01:09:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.507+0000 7f8388ff9700 1 -- 172.21.15.39:0/3575684745 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f838400f040 con 0x7f838c0b2510 2023-06-23T01:09:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.507+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837c009420 con 0x7f838c0b2510 2023-06-23T01:09:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.507+0000 7f8388ff9700 1 -- 172.21.15.39:0/3575684745 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83840146b0 con 0x7f838c0b2510 2023-06-23T01:09:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.507+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838c15c2d0 con 0x7f838c0b2510 2023-06-23T01:09:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.508+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838c0ce610 con 0x7f838c0b2510 2023-06-23T01:09:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.508+0000 7f8388ff9700 1 -- 172.21.15.39:0/3575684745 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f8384014a20 con 0x7f838c0b2510 2023-06-23T01:09:00.516 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.509+0000 7f8388ff9700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8378066e90 0x7f8378069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:00.517 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.509+0000 7f8388ff9700 1 -- 172.21.15.39:0/3575684745 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8384084f20 con 0x7f838c0b2510 2023-06-23T01:09:00.517 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.509+0000 7f838bfff700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8378066e90 0x7f8378069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:00.517 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.509+0000 7f838bfff700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8378066e90 0x7f8378069340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f838c154260 tx=0x7f8380009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:00.518 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.513+0000 7f8388ff9700 1 -- 172.21.15.39:0/3575684745 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f8384052440 con 0x7f838c0b2510 2023-06-23T01:09:00.707 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:00.704+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f838c04c1b0 con 0x7f838c0b2510 2023-06-23T01:09:01.701 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.698+0000 7f8388ff9700 1 -- 172.21.15.39:0/3575684745 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f8384055a60 con 0x7f838c0b2510 2023-06-23T01:09:01.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8378066e90 msgr2=0x7f8378069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:01.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8378066e90 0x7f8378069340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f838c154260 tx=0x7f8380009040 comp rx=0 tx=0).stop 2023-06-23T01:09:01.705 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f838c0b2510 msgr2=0x7f838c14c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:01.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f838c0b2510 0x7f838c14c2b0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f838400c1a0 tx=0x7f838400b040 comp rx=0 tx=0).stop 2023-06-23T01:09:01.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 shutdown_connections 2023-06-23T01:09:01.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f838c153ce0 0x7f838c15b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:01.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f838c0b2510 0x7f838c14c2b0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:01.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f8378066e90 0x7f8378069340 secure :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f838c154260 tx=0x7f8380009040 comp rx=0 tx=0).stop 2023-06-23T01:09:01.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 --2- 172.21.15.39:0/3575684745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f838c153910 0x7f838c154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:01.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 >> 172.21.15.39:0/3575684745 conn(0x7f838c067450 msgr2=0x7f838c0b4030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:01.707 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 shutdown_connections 2023-06-23T01:09:01.707 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.702+0000 7f8392f21700 1 -- 172.21.15.39:0/3575684745 wait complete. 2023-06-23T01:09:01.707 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_metadata' created 2023-06-23T01:09:01.720 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-23T01:09:01.848 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.845+0000 7f0c372fd700 1 Processor -- start 2023-06-23T01:09:01.848 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.845+0000 7f0c372fd700 1 -- start start 2023-06-23T01:09:01.850 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c372fd700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159910 0x7f0c300b0f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:01.851 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c372fd700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c3015db30 con 0x7f0c30159540 2023-06-23T01:09:01.851 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c372fd700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c30068bb0 con 0x7f0c3006a390 2023-06-23T01:09:01.851 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c372fd700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3015e330 con 0x7f0c30159910 2023-06-23T01:09:01.851 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c35099700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f0c30159540 0x7f0c3015d420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52636/0 (socket says 172.21.15.39:52636) 2023-06-23T01:09:01.851 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c35099700 1 -- 172.21.15.39:0/4215428710 learned_addr learned my addr 172.21.15.39:0/4215428710 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:09:01.851 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c3589a700 1 --2- 172.21.15.39:0/4215428710 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159910 0x7f0c300b0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:01.852 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c2ffff700 1 -- 172.21.15.39:0/4215428710 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208678166 0 0) 0x7f0c20001490 con 0x7f0c30159540 2023-06-23T01:09:01.852 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c2ffff700 1 -- 172.21.15.39:0/4215428710 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c18003570 con 0x7f0c30159540 2023-06-23T01:09:01.852 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c2ffff700 1 -- 172.21.15.39:0/4215428710 <== mon.1 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3383500867 0 0) 0x7f0c30068bb0 con 0x7f0c3006a390 2023-06-23T01:09:01.852 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c2ffff700 1 -- 172.21.15.39:0/4215428710 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c20001490 con 0x7f0c3006a390 2023-06-23T01:09:01.852 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c3589a700 1 -- 172.21.15.39:0/4215428710 >> v1:172.21.15.73:6789/0 conn(0x7f0c3006a390 legacy=0x7f0c30069040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:01.852 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c3589a700 1 -- 172.21.15.39:0/4215428710 >> v1:172.21.15.39:6789/0 conn(0x7f0c30159540 legacy=0x7f0c3015d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:01.853 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.846+0000 7f0c3589a700 1 -- 172.21.15.39:0/4215428710 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3015dd70 con 0x7f0c30159910 2023-06-23T01:09:01.853 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c3589a700 1 --2- 172.21.15.39:0/4215428710 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159910 0x7f0c300b0f70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009910 tx=0x7f0c280091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3b835a656b6cd3b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:01.853 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c2ffff700 1 -- 172.21.15.39:0/4215428710 <== mon.2 v2:172.21.15.73:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28011630 con 0x7f0c30159910 2023-06-23T01:09:01.853 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c2ffff700 1 -- 172.21.15.39:0/4215428710 <== mon.2 v2:172.21.15.73:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28011790 con 0x7f0c30159910 2023-06-23T01:09:01.854 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c2ffff700 1 -- 172.21.15.39:0/4215428710 <== mon.2 v2:172.21.15.73:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c28011b30 con 0x7f0c30159910 2023-06-23T01:09:01.854 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c372fd700 1 -- 172.21.15.39:0/4215428710 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159910 msgr2=0x7f0c300b0f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:01.854 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c372fd700 1 --2- 172.21.15.39:0/4215428710 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159910 0x7f0c300b0f70 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0c28009910 tx=0x7f0c280091a0 comp rx=0 tx=0).stop 2023-06-23T01:09:01.855 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c372fd700 1 -- 172.21.15.39:0/4215428710 shutdown_connections 2023-06-23T01:09:01.855 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c372fd700 1 --2- 172.21.15.39:0/4215428710 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159910 0x7f0c300b0f70 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:01.855 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c372fd700 1 -- 172.21.15.39:0/4215428710 >> 172.21.15.39:0/4215428710 conn(0x7f0c30070160 msgr2=0x7f0c30070560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:01.855 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.847+0000 7f0c372fd700 1 -- 172.21.15.39:0/4215428710 shutdown_connections 2023-06-23T01:09:01.855 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.848+0000 7f0c372fd700 1 -- 172.21.15.39:0/4215428710 wait complete. 2023-06-23T01:09:01.857 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.849+0000 7f0c372fd700 1 Processor -- start 2023-06-23T01:09:01.857 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.849+0000 7f0c372fd700 1 -- start start 2023-06-23T01:09:01.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.849+0000 7f0c372fd700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159540 0x7f0c3015ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:01.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.849+0000 7f0c372fd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c30159910 0x7f0c30150970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:01.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.849+0000 7f0c372fd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c300b1610 0x7f0c300c79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:01.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.849+0000 7f0c372fd700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c30150eb0 con 0x7f0c30159910 2023-06-23T01:09:01.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.849+0000 7f0c372fd700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c300c2a80 con 0x7f0c300b1610 2023-06-23T01:09:01.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.849+0000 7f0c372fd700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3015aaa0 con 0x7f0c30159540 2023-06-23T01:09:01.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c30159910 0x7f0c30150970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:01.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c30159910 0x7f0c30150970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58420/0 (socket says 172.21.15.39:58420) 2023-06-23T01:09:01.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 -- 172.21.15.39:0/2214632840 learned_addr learned my addr 172.21.15.39:0/2214632840 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:01.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c35099700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159540 0x7f0c3015ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:01.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c3589a700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c300b1610 0x7f0c300c79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:01.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 -- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159540 msgr2=0x7f0c3015ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:01.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159540 0x7f0c3015ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:01.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 -- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c300b1610 msgr2=0x7f0c300c79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:01.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c300b1610 0x7f0c300c79d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:01.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 -- 172.21.15.39:0/2214632840 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c28009420 con 0x7f0c30159910 2023-06-23T01:09:01.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c35099700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159540 0x7f0c3015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:09:01.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c3589a700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c300b1610 0x7f0c300c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:09:01.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c34898700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c30159910 0x7f0c30150970 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400c7f0 tx=0x7f0c24007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:01.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c2d7fa700 1 -- 172.21.15.39:0/2214632840 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24010510 con 0x7f0c30159910 2023-06-23T01:09:01.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c2d7fa700 1 -- 172.21.15.39:0/2214632840 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c24010670 con 0x7f0c30159910 2023-06-23T01:09:01.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c2d7fa700 1 -- 172.21.15.39:0/2214632840 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c24010a30 con 0x7f0c30159910 2023-06-23T01:09:01.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c300c2cf0 con 0x7f0c30159910 2023-06-23T01:09:01.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.850+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3015a770 con 0x7f0c30159910 2023-06-23T01:09:01.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.851+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c3004c1b0 con 0x7f0c30159910 2023-06-23T01:09:01.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.852+0000 7f0c2d7fa700 1 -- 172.21.15.39:0/2214632840 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f0c2401b400 con 0x7f0c30159910 2023-06-23T01:09:01.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.852+0000 7f0c2d7fa700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0c1c066e90 0x7f0c1c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:01.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.852+0000 7f0c2d7fa700 1 -- 172.21.15.39:0/2214632840 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f0c240840c0 con 0x7f0c30159910 2023-06-23T01:09:01.862 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.853+0000 7f0c35099700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0c1c066e90 0x7f0c1c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:01.862 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.853+0000 7f0c35099700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0c1c066e90 0x7f0c1c069340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c20005c50 tx=0x7f0c20009040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:01.862 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:01.857+0000 7f0c2d7fa700 1 -- 172.21.15.39:0/2214632840 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f0c24051470 con 0x7f0c30159910 2023-06-23T01:09:02.053 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.050+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f0c300c3210 con 0x7f0c30159910 2023-06-23T01:09:02.706 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.702+0000 7f0c2d7fa700 1 -- 172.21.15.39:0/2214632840 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v21) v1 ==== 170+0+0 (secure 0 0 0) 0x7f0c300c3210 con 0x7f0c30159910 2023-06-23T01:09:02.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.707+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0c1c066e90 msgr2=0x7f0c1c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:02.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.707+0000 7f0c372fd700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0c1c066e90 0x7f0c1c069340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0c20005c50 tx=0x7f0c20009040 comp rx=0 tx=0).stop 2023-06-23T01:09:02.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.707+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c30159910 msgr2=0x7f0c30150970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:02.711 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.707+0000 7f0c372fd700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c30159910 0x7f0c30150970 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0c2400c7f0 tx=0x7f0c24007dc0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.708+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 shutdown_connections 2023-06-23T01:09:02.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.708+0000 7f0c372fd700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f0c1c066e90 0x7f0c1c069340 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.708+0000 7f0c372fd700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c300b1610 0x7f0c300c79d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.708+0000 7f0c372fd700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f0c30159540 0x7f0c3015ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.708+0000 7f0c372fd700 1 --2- 172.21.15.39:0/2214632840 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f0c30159910 0x7f0c30150970 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.712 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.708+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 >> 172.21.15.39:0/2214632840 conn(0x7f0c30070160 msgr2=0x7f0c30156a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:02.713 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.710+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 shutdown_connections 2023-06-23T01:09:02.714 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.711+0000 7f0c372fd700 1 -- 172.21.15.39:0/2214632840 wait complete. 2023-06-23T01:09:02.720 INFO:teuthology.orchestra.run.smithi039.stderr:pool 'cephfs_data' created 2023-06-23T01:09:02.733 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-23T01:09:02.856 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.853+0000 7f63d1e22700 1 Processor -- start 2023-06-23T01:09:02.856 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.853+0000 7f63d1e22700 1 -- start start 2023-06-23T01:09:02.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63d1e22700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc1579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:02.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63d1e22700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63cc15af60 0x7f63cc15d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:02.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63d1e22700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63cc15c0e0 con 0x7f63cc0b2af0 2023-06-23T01:09:02.858 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63d1e22700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63cc15b4d0 con 0x7f63cc15af60 2023-06-23T01:09:02.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63d1e22700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc15b7b0 con 0x7f63cc0b2720 2023-06-23T01:09:02.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cbfff700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63cc15af60 0x7f63cc15d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:02.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cbfff700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7f63cc15af60 0x7f63cc15d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58430/0 (socket says 172.21.15.39:58430) 2023-06-23T01:09:02.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cbfff700 1 -- 172.21.15.39:0/3136307695 learned_addr learned my addr 172.21.15.39:0/3136307695 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:02.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cb7fe700 1 --2- 172.21.15.39:0/3136307695 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc1579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:02.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cbfff700 1 -- 172.21.15.39:0/3136307695 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 msgr2=0x7f63cc1579d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:02.859 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cbfff700 1 --2- 172.21.15.39:0/3136307695 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc1579d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cbfff700 1 -- 172.21.15.39:0/3136307695 >> v1:172.21.15.73:6789/0 conn(0x7f63cc0b2af0 legacy=0x7f63cc0cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:02.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cbfff700 1 -- 172.21.15.39:0/3136307695 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63cc15ba90 con 0x7f63cc15af60 2023-06-23T01:09:02.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cb7fe700 1 --2- 172.21.15.39:0/3136307695 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc1579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:09:02.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63cbfff700 1 --2- 172.21.15.39:0/3136307695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc15d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009f20 tx=0x7f63c4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292ff89d74172393 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:02.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63ca7fc700 1 -- 172.21.15.39:0/3136307695 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c400e070 con 0x7f63cc15af60 2023-06-23T01:09:02.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63ca7fc700 1 -- 172.21.15.39:0/3136307695 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c400ba40 con 0x7f63cc15af60 2023-06-23T01:09:02.860 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.854+0000 7f63ca7fc700 1 -- 172.21.15.39:0/3136307695 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c40092e0 con 0x7f63cc15af60 2023-06-23T01:09:02.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.855+0000 7f63d1e22700 1 -- 172.21.15.39:0/3136307695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 msgr2=0x7f63cc15d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:02.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.855+0000 7f63d1e22700 1 --2- 172.21.15.39:0/3136307695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc15d390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f63c4009f20 tx=0x7f63c4009840 comp rx=0 tx=0).stop 2023-06-23T01:09:02.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.855+0000 7f63d1e22700 1 -- 172.21.15.39:0/3136307695 shutdown_connections 2023-06-23T01:09:02.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.855+0000 7f63d1e22700 1 --2- 172.21.15.39:0/3136307695 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc15d390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.855+0000 7f63d1e22700 1 --2- 172.21.15.39:0/3136307695 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc1579d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.861 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.855+0000 7f63d1e22700 1 -- 172.21.15.39:0/3136307695 >> 172.21.15.39:0/3136307695 conn(0x7f63cc067450 msgr2=0x7f63cc067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:02.862 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.855+0000 7f63d1e22700 1 -- 172.21.15.39:0/3136307695 shutdown_connections 2023-06-23T01:09:02.862 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.855+0000 7f63d1e22700 1 -- 172.21.15.39:0/3136307695 wait complete. 2023-06-23T01:09:02.864 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.856+0000 7f63d1e22700 1 Processor -- start 2023-06-23T01:09:02.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.856+0000 7f63d1e22700 1 -- start start 2023-06-23T01:09:02.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63d1e22700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc0d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:02.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63d1e22700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63cc0b2af0 0x7f63cc14d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:02.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63d1e22700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc0c61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:02.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63d1e22700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0b4430 con 0x7f63cc15af60 2023-06-23T01:09:02.865 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63d1e22700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0b3820 con 0x7f63cc0b2af0 2023-06-23T01:09:02.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63d1e22700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63cc0b3b00 con 0x7f63cc0b2720 2023-06-23T01:09:02.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc0c61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:02.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc0c61b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58436/0 (socket says 172.21.15.39:58436) 2023-06-23T01:09:02.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cb7fe700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc0d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:02.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 -- 172.21.15.39:0/1728214303 learned_addr learned my addr 172.21.15.39:0/1728214303 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:02.866 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63caffd700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63cc0b2af0 0x7f63cc14d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:02.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 -- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 msgr2=0x7f63cc0d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:02.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc0d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 -- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63cc0b2af0 msgr2=0x7f63cc14d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:02.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63cc0b2af0 0x7f63cc14d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:02.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 -- 172.21.15.39:0/1728214303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c4009ae0 con 0x7f63cc15af60 2023-06-23T01:09:02.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.857+0000 7f63cbfff700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc0c61b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63cc065560 tx=0x7f63c4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:02.867 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.858+0000 7f63c37fe700 1 -- 172.21.15.39:0/1728214303 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c40094e0 con 0x7f63cc15af60 2023-06-23T01:09:02.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.858+0000 7f63c37fe700 1 -- 172.21.15.39:0/1728214303 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c40032f0 con 0x7f63cc15af60 2023-06-23T01:09:02.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.858+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63cc0b3de0 con 0x7f63cc15af60 2023-06-23T01:09:02.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.858+0000 7f63c37fe700 1 -- 172.21.15.39:0/1728214303 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c401d420 con 0x7f63cc15af60 2023-06-23T01:09:02.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.858+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63cc0d1430 con 0x7f63cc15af60 2023-06-23T01:09:02.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.859+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63cc0b45e0 con 0x7f63cc15af60 2023-06-23T01:09:02.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.859+0000 7f63c37fe700 1 -- 172.21.15.39:0/1728214303 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f63c401d850 con 0x7f63cc15af60 2023-06-23T01:09:02.868 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.859+0000 7f63c37fe700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f63ac066e90 0x7f63ac069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:02.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.860+0000 7f63c37fe700 1 -- 172.21.15.39:0/1728214303 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f63c4085c20 con 0x7f63cc15af60 2023-06-23T01:09:02.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.860+0000 7f63cb7fe700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f63ac066e90 0x7f63ac069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:02.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.860+0000 7f63cb7fe700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f63ac066e90 0x7f63ac069340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00adc0 tx=0x7f63bc006f90 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:02.869 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:02.864+0000 7f63c37fe700 1 -- 172.21.15.39:0/1728214303 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f63cc0b45e0 con 0x7f63cc15af60 2023-06-23T01:09:03.105 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.102+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f63cc04c1b0 con 0x7f63cc15af60 2023-06-23T01:09:03.110 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.107+0000 7f63c37fe700 1 -- 172.21.15.39:0/1728214303 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f63c4052e50 con 0x7f63cc15af60 2023-06-23T01:09:03.118 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f63ac066e90 msgr2=0x7f63ac069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.119 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f63ac066e90 0x7f63ac069340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f63bc00adc0 tx=0x7f63bc006f90 comp rx=0 tx=0).stop 2023-06-23T01:09:03.119 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 msgr2=0x7f63cc0c61b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.119 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc0c61b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f63cc065560 tx=0x7f63c4017000 comp rx=0 tx=0).stop 2023-06-23T01:09:03.119 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 shutdown_connections 2023-06-23T01:09:03.119 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f63cc15af60 0x7f63cc0c61b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.119 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f63cc0b2720 0x7f63cc0d0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.120 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f63ac066e90 0x7f63ac069340 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.120 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 --2- 172.21.15.39:0/1728214303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f63cc0b2af0 0x7f63cc14d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.120 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 >> 172.21.15.39:0/1728214303 conn(0x7f63cc067450 msgr2=0x7f63cc157960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:03.120 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 shutdown_connections 2023-06-23T01:09:03.120 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.115+0000 7f63d1e22700 1 -- 172.21.15.39:0/1728214303 wait complete. 2023-06-23T01:09:03.132 INFO:teuthology.orchestra.run.smithi039.stderr:new fs with metadata pool 3 and data pool 4 2023-06-23T01:09:03.141 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:09:03.259 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.256+0000 7fd37a6a9700 1 Processor -- start 2023-06-23T01:09:03.260 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.256+0000 7fd37a6a9700 1 -- start start 2023-06-23T01:09:03.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd37a6a9700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd3740c7760 0x7fd3740cacf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd37a6a9700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740cb230 0x7fd3740c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd37a6a9700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3740cb690 con 0x7fd374142f50 2023-06-23T01:09:03.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd37a6a9700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3740c88e0 con 0x7fd3740c7760 2023-06-23T01:09:03.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd37a6a9700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd374143320 con 0x7fd3740cb230 2023-06-23T01:09:03.262 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd3737fe700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7fd3740c7760 0x7fd3740cacf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.263 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd373fff700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7fd374142f50 0x7fd374068750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52642/0 (socket says 172.21.15.39:52642) 2023-06-23T01:09:03.263 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd373fff700 1 -- 172.21.15.39:0/2657501592 learned_addr learned my addr 172.21.15.39:0/2657501592 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:09:03.263 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.257+0000 7fd378c46700 1 --2- 172.21.15.39:0/2657501592 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740cb230 0x7fd3740c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.263 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd3737fe700 1 -- 172.21.15.39:0/2657501592 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740cb230 msgr2=0x7fd3740c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:03.263 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd3737fe700 1 --2- 172.21.15.39:0/2657501592 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740cb230 0x7fd3740c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.263 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd3737fe700 1 -- 172.21.15.39:0/2657501592 >> v1:172.21.15.39:6789/0 conn(0x7fd374142f50 legacy=0x7fd374068750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.263 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd372ffd700 1 -- 172.21.15.39:0/2657501592 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1299417394 0 0) 0x7fd364001110 con 0x7fd374142f50 2023-06-23T01:09:03.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd3737fe700 1 -- 172.21.15.39:0/2657501592 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3740c7cd0 con 0x7fd3740c7760 2023-06-23T01:09:03.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd3737fe700 1 --2- 172.21.15.39:0/2657501592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740c7760 0x7fd3740cacf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd3680098e0 tx=0x7fd3680091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4eda3e28791a176 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd372ffd700 1 -- 172.21.15.39:0/2657501592 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd368009e80 con 0x7fd3740c7760 2023-06-23T01:09:03.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd372ffd700 1 -- 172.21.15.39:0/2657501592 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd36800b860 con 0x7fd3740c7760 2023-06-23T01:09:03.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd372ffd700 1 -- 172.21.15.39:0/2657501592 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36800bbe0 con 0x7fd3740c7760 2023-06-23T01:09:03.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd37a6a9700 1 -- 172.21.15.39:0/2657501592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740c7760 msgr2=0x7fd3740cacf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.264 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/2657501592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740c7760 0x7fd3740cacf0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd3680098e0 tx=0x7fd3680091a0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd37a6a9700 1 -- 172.21.15.39:0/2657501592 shutdown_connections 2023-06-23T01:09:03.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/2657501592 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740cb230 0x7fd3740c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/2657501592 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740c7760 0x7fd3740cacf0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.258+0000 7fd37a6a9700 1 -- 172.21.15.39:0/2657501592 >> 172.21.15.39:0/2657501592 conn(0x7fd37406b250 msgr2=0x7fd37406b650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:03.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.259+0000 7fd37a6a9700 1 -- 172.21.15.39:0/2657501592 shutdown_connections 2023-06-23T01:09:03.265 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.259+0000 7fd37a6a9700 1 -- 172.21.15.39:0/2657501592 wait complete. 2023-06-23T01:09:03.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.260+0000 7fd37a6a9700 1 Processor -- start 2023-06-23T01:09:03.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.260+0000 7fd37a6a9700 1 -- start start 2023-06-23T01:09:03.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.260+0000 7fd37a6a9700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740c7760 0x7fd374158b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.267 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.260+0000 7fd37a6a9700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740cb230 0x7fd374159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.260+0000 7fd37a6a9700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd374142f50 0x7fd37415a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.260+0000 7fd37a6a9700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37415a820 con 0x7fd3740cb230 2023-06-23T01:09:03.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.260+0000 7fd37a6a9700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd37412fac0 con 0x7fd374142f50 2023-06-23T01:09:03.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.260+0000 7fd37a6a9700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd37412f770 con 0x7fd3740c7760 2023-06-23T01:09:03.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740cb230 0x7fd374159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740cb230 0x7fd374159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58460/0 (socket says 172.21.15.39:58460) 2023-06-23T01:09:03.268 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 -- 172.21.15.39:0/3365726613 learned_addr learned my addr 172.21.15.39:0/3365726613 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd378c46700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd374142f50 0x7fd37415a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 -- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740c7760 msgr2=0x7fd374158b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740c7760 0x7fd374158b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 -- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd374142f50 msgr2=0x7fd37415a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd374142f50 0x7fd37415a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 -- 172.21.15.39:0/3365726613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd368009420 con 0x7fd3740cb230 2023-06-23T01:09:03.269 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd3737fe700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740cb230 0x7fd374159da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd368013000 tx=0x7fd368017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd35bfff700 1 -- 172.21.15.39:0/3365726613 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd36800bbe0 con 0x7fd3740cb230 2023-06-23T01:09:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd35bfff700 1 -- 172.21.15.39:0/3365726613 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd368011420 con 0x7fd3740cb230 2023-06-23T01:09:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3740c6e00 con 0x7fd3740cb230 2023-06-23T01:09:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd35bfff700 1 -- 172.21.15.39:0/3365726613 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3680117e0 con 0x7fd3740cb230 2023-06-23T01:09:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.261+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3740c73c0 con 0x7fd3740cb230 2023-06-23T01:09:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.262+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd37404c1b0 con 0x7fd3740cb230 2023-06-23T01:09:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.262+0000 7fd35bfff700 1 -- 172.21.15.39:0/3365726613 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fd368011bd0 con 0x7fd3740cb230 2023-06-23T01:09:03.270 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.263+0000 7fd35bfff700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd360066ec0 0x7fd360069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.271 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.263+0000 7fd35bfff700 1 -- 172.21.15.39:0/3365726613 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fd368085f00 con 0x7fd3740cb230 2023-06-23T01:09:03.271 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.263+0000 7fd373fff700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd360066ec0 0x7fd360069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.271 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.263+0000 7fd373fff700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd360066ec0 0x7fd360069370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3640016e0 tx=0x7fd364001260 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.271 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.267+0000 7fd35bfff700 1 -- 172.21.15.39:0/3365726613 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fd3680530d0 con 0x7fd3740cb230 2023-06-23T01:09:03.464 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.461+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3740c7fb0 con 0x7fd3740cb230 2023-06-23T01:09:03.465 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.462+0000 7fd35bfff700 1 -- 172.21.15.39:0/3365726613 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16927 (secure 0 0 0) 0x7fd368015090 con 0x7fd3740cb230 2023-06-23T01:09:03.466 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:09:03.466 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"314a4b16-8059-4f2c-986c-1624c6de1afb","created":"2023-06-23T01:08:37.443723+0000","modified":"2023-06-23T01:09:03.103682+0000","last_up_change":"2023-06-23T01:08:48.497542+0000","last_in_change":"2023-06-23T01:08:40.002994+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:08:49.048645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:08:55.930040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:09:00.734607+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T01:09:02.080383+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a33e557d-f70e-4a1c-a10e-bc7f351dee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":117964},{"type":"v1","addr":"172.21.15.39:6817","nonce":117964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":117964},{"type":"v1","addr":"172.21.15.39:6819","nonce":117964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":117964},{"type":"v1","addr":"172.21.15.39:6823","nonce":117964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":117964},{"type":"v1","addr":"172.21.15.39:6821","nonce":117964}]},"public_addr":"172.21.15.39:6817/117964","cluster_addr":"172.21.15.39:6819/117964","heartbeat_back_addr":"172.21.15.39:6823/117964","heartbeat_front_addr":"172.21.15.39:6821/117964","state":["exists","up"]},{"osd":1,"uuid":"757696fc-3c60-4f7f-822d-a8ff1e89534d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":117963},{"type":"v1","addr":"172.21.15.39:6801","nonce":117963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":117963},{"type":"v1","addr":"172.21.15.39:6803","nonce":117963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":117963},{"type":"v1","addr":"172.21.15.39:6807","nonce":117963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":117963},{"type":"v1","addr":"172.21.15.39:6805","nonce":117963}]},"public_addr":"172.21.15.39:6801/117963","cluster_addr":"172.21.15.39:6803/117963","heartbeat_back_addr":"172.21.15.39:6807/117963","heartbeat_front_addr":"172.21.15.39:6805/117963","state":["exists","up"]},{"osd":2,"uuid":"cffcd6d2-b414-40b0-8967-26421b058b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":117974},{"type":"v1","addr":"172.21.15.39:6825","nonce":117974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":117974},{"type":"v1","addr":"172.21.15.39:6827","nonce":117974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":117974},{"type":"v1","addr":"172.21.15.39:6831","nonce":117974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":117974},{"type":"v1","addr":"172.21.15.39:6829","nonce":117974}]},"public_addr":"172.21.15.39:6825/117974","cluster_addr":"172.21.15.39:6827/117974","heartbeat_back_addr":"172.21.15.39:6831/117974","heartbeat_front_addr":"172.21.15.39:6829/117974","state":["exists","up"]},{"osd":3,"uuid":"5a91897e-5924-4d57-ac5e-bfd06cf4ff93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":117973},{"type":"v1","addr":"172.21.15.39:6809","nonce":117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":117973},{"type":"v1","addr":"172.21.15.39:6811","nonce":117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":117973},{"type":"v1","addr":"172.21.15.39:6815","nonce":117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":117973},{"type":"v1","addr":"172.21.15.39:6813","nonce":117973}]},"public_addr":"172.21.15.39:6809/117973","cluster_addr":"172.21.15.39:6811/117973","heartbeat_back_addr":"172.21.15.39:6815/117973","heartbeat_front_addr":"172.21.15.39:6813/117973","state":["exists","up"]},{"osd":4,"uuid":"67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":118146},{"type":"v1","addr":"172.21.15.73:6801","nonce":118146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":118146},{"type":"v1","addr":"172.21.15.73:6803","nonce":118146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":118146},{"type":"v1","addr":"172.21.15.73:6807","nonce":118146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":118146},{"type":"v1","addr":"172.21.15.73:6805","nonce":118146}]},"public_addr":"172.21.15.73:6801/118146","cluster_addr":"172.21.15.73:6803/118146","heartbeat_back_addr":"172.21.15.73:6807/118146","heartbeat_front_addr":"172.21.15.73:6805/118146","state":["exists","up"]},{"osd":5,"uuid":"390058ef-ef47-475e-af72-8e2f51ee184c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":118148},{"type":"v1","addr":"172.21.15.73:6817","nonce":118148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":118148},{"type":"v1","addr":"172.21.15.73:6819","nonce":118148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":118148},{"type":"v1","addr":"172.21.15.73:6823","nonce":118148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":118148},{"type":"v1","addr":"172.21.15.73:6821","nonce":118148}]},"public_addr":"172.21.15.73:6817/118148","cluster_addr":"172.21.15.73:6819/118148","heartbeat_back_addr":"172.21.15.73:6823/118148","heartbeat_front_addr":"172.21.15.73:6821/118148","state":["exists","up"]},{"osd":6,"uuid":"c34ad563-5018-4cf4-830f-2ac56a3a5e5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":118147},{"type":"v1","addr":"172.21.15.73:6809","nonce":118147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":118147},{"type":"v1","addr":"172.21.15.73:6811","nonce":118147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":118147},{"type":"v1","addr":"172.21.15.73:6815","nonce":118147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":118147},{"type":"v1","addr":"172.21.15.73:6813","nonce":118147}]},"public_addr":"172.21.15.73:6809/118147","cluster_addr":"172.21.15.73:6811/118147","heartbeat_back_addr":"172.21.15.73:6815/118147","heartbeat_front_addr":"172.21.15.73:6813/118147","state":["exists","up"]},{"osd":7,"uuid":"e2023d49-2e44-412c-8cd1-c67447567c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":118189},{"type":"v1","addr":"172.21.15.73:6825","nonce":118189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":118189},{"type":"v1","addr":"172.21.15.73:6827","nonce":118189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":118189},{"type":"v1","addr":"172.21.15.73:6831","nonce":118189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":118189},{"type":"v1","addr":"172.21.15.73:6829","nonce":118189}]},"public_addr":"172.21.15.73:6825/118189","cluster_addr":"172.21.15.73:6827/118189","heartbeat_back_addr":"172.21.15.73:6831/118189","heartbeat_front_addr":"172.21.15.73:6829/118189","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.198991+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.140721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.167693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.101533+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.095800+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.189388+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.133372+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.190656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:09:03.468 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd360066ec0 msgr2=0x7fd360069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.468 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd360066ec0 0x7fd360069370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd3640016e0 tx=0x7fd364001260 comp rx=0 tx=0).stop 2023-06-23T01:09:03.468 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740cb230 msgr2=0x7fd374159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740cb230 0x7fd374159da0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd368013000 tx=0x7fd368017000 comp rx=0 tx=0).stop 2023-06-23T01:09:03.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 shutdown_connections 2023-06-23T01:09:03.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7fd360066ec0 0x7fd360069370 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.469 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd374142f50 0x7fd37415a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7fd3740c7760 0x7fd374158b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 --2- 172.21.15.39:0/3365726613 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7fd3740cb230 0x7fd374159da0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.470 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 >> 172.21.15.39:0/3365726613 conn(0x7fd37406b250 msgr2=0x7fd374141f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:03.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 shutdown_connections 2023-06-23T01:09:03.471 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.464+0000 7fd37a6a9700 1 -- 172.21.15.39:0/3365726613 wait complete. 2023-06-23T01:09:03.480 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:09:03.598 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.595+0000 7ff64dbbc700 1 Processor -- start 2023-06-23T01:09:03.598 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.595+0000 7ff64dbbc700 1 -- start start 2023-06-23T01:09:03.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff64dbbc700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6480b2720 0x7ff6481579d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.601 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff64dbbc700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2af0 0x7ff6480cd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff64dbbc700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7ff64815af60 0x7ff64815d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff64dbbc700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64815c0e0 con 0x7ff6480b2720 2023-06-23T01:09:03.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff64dbbc700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff64815b4d0 con 0x7ff64815af60 2023-06-23T01:09:03.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff64dbbc700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff64815b7b0 con 0x7ff6480b2af0 2023-06-23T01:09:03.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff6477fe700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6480b2720 0x7ff6481579d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff6477fe700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff6480b2720 0x7ff6481579d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58468/0 (socket says 172.21.15.39:58468) 2023-06-23T01:09:03.602 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff6477fe700 1 -- 172.21.15.39:0/1718475735 learned_addr learned my addr 172.21.15.39:0/1718475735 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:03.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff647fff700 1 --2- 172.21.15.39:0/1718475735 >> v2:172.21.15.73:3300/0 conn(0x7ff64815af60 0x7ff64815d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff646ffd700 1 --2- 172.21.15.39:0/1718475735 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2af0 0x7ff6480cd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff6477fe700 1 -- 172.21.15.39:0/1718475735 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2af0 msgr2=0x7ff6480cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:03.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.596+0000 7ff6477fe700 1 --2- 172.21.15.39:0/1718475735 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2af0 0x7ff6480cd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff6477fe700 1 -- 172.21.15.39:0/1718475735 >> v2:172.21.15.73:3300/0 conn(0x7ff64815af60 msgr2=0x7ff64815d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:03.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff6477fe700 1 --2- 172.21.15.39:0/1718475735 >> v2:172.21.15.73:3300/0 conn(0x7ff64815af60 0x7ff64815d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff6477fe700 1 -- 172.21.15.39:0/1718475735 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64815ba90 con 0x7ff6480b2720 2023-06-23T01:09:03.603 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff6477fe700 1 --2- 172.21.15.39:0/1718475735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2720 0x7ff6481579d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6480caa10 tx=0x7ff638009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5c24cb5f76e163d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff6467fc700 1 -- 172.21.15.39:0/1718475735 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63800b640 con 0x7ff6480b2720 2023-06-23T01:09:03.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff6467fc700 1 -- 172.21.15.39:0/1718475735 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63800b7a0 con 0x7ff6480b2720 2023-06-23T01:09:03.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff6467fc700 1 -- 172.21.15.39:0/1718475735 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63800bb60 con 0x7ff6480b2720 2023-06-23T01:09:03.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 -- 172.21.15.39:0/1718475735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2720 msgr2=0x7ff6481579d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/1718475735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2720 0x7ff6481579d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6480caa10 tx=0x7ff638009250 comp rx=0 tx=0).stop 2023-06-23T01:09:03.604 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 -- 172.21.15.39:0/1718475735 shutdown_connections 2023-06-23T01:09:03.605 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/1718475735 >> v2:172.21.15.73:3300/0 conn(0x7ff64815af60 0x7ff64815d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.605 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/1718475735 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2720 0x7ff6481579d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.605 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/1718475735 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2af0 0x7ff6480cd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.605 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 -- 172.21.15.39:0/1718475735 >> 172.21.15.39:0/1718475735 conn(0x7ff648067450 msgr2=0x7ff648067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:03.605 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 -- 172.21.15.39:0/1718475735 shutdown_connections 2023-06-23T01:09:03.605 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.597+0000 7ff64dbbc700 1 -- 172.21.15.39:0/1718475735 wait complete. 2023-06-23T01:09:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff64dbbc700 1 Processor -- start 2023-06-23T01:09:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff64dbbc700 1 -- start start 2023-06-23T01:09:03.610 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff64dbbc700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2720 0x7ff6480c74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff64dbbc700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2af0 0x7ff6480c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff64dbbc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff64815af60 0x7ff6480c9520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff64dbbc700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480ce980 con 0x7ff6480b2af0 2023-06-23T01:09:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff64dbbc700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480cec60 con 0x7ff64815af60 2023-06-23T01:09:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff64dbbc700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6480cef40 con 0x7ff6480b2720 2023-06-23T01:09:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.599+0000 7ff646ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2af0 0x7ff6480c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.611 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff646ffd700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2af0 0x7ff6480c7a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58474/0 (socket says 172.21.15.39:58474) 2023-06-23T01:09:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff646ffd700 1 -- 172.21.15.39:0/3974019488 learned_addr learned my addr 172.21.15.39:0/3974019488 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff647fff700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff64815af60 0x7ff6480c9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff6477fe700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2720 0x7ff6480c74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff646ffd700 1 -- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2720 msgr2=0x7ff6480c74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff646ffd700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2720 0x7ff6480c74f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.612 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff646ffd700 1 -- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff64815af60 msgr2=0x7ff6480c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff646ffd700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff64815af60 0x7ff6480c9520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff646ffd700 1 -- 172.21.15.39:0/3974019488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6380094f0 con 0x7ff6480b2af0 2023-06-23T01:09:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff646ffd700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2af0 0x7ff6480c7a30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00ec80 tx=0x7ff63c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff62f7fe700 1 -- 172.21.15.39:0/3974019488 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c00c850 con 0x7ff6480b2af0 2023-06-23T01:09:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff62f7fe700 1 -- 172.21.15.39:0/3974019488 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff63c00c9b0 con 0x7ff6480b2af0 2023-06-23T01:09:03.613 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6480cf220 con 0x7ff6480b2af0 2023-06-23T01:09:03.614 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff62f7fe700 1 -- 172.21.15.39:0/3974019488 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff63c00cd50 con 0x7ff6480b2af0 2023-06-23T01:09:03.614 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.600+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6480c9a60 con 0x7ff6480b2af0 2023-06-23T01:09:03.614 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.601+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6480cf6c0 con 0x7ff6480b2af0 2023-06-23T01:09:03.614 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.601+0000 7ff62f7fe700 1 -- 172.21.15.39:0/3974019488 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7ff63c013680 con 0x7ff6480b2af0 2023-06-23T01:09:03.614 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.602+0000 7ff62f7fe700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff634066e90 0x7ff634069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.614 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.602+0000 7ff62f7fe700 1 -- 172.21.15.39:0/3974019488 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff63c0847b0 con 0x7ff6480b2af0 2023-06-23T01:09:03.614 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.602+0000 7ff6477fe700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff634066e90 0x7ff634069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.602+0000 7ff6477fe700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff634066e90 0x7ff634069340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff6480caa10 tx=0x7ff638017000 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.615 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.606+0000 7ff62f7fe700 1 -- 172.21.15.39:0/3974019488 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7ff63c051980 con 0x7ff6480b2af0 2023-06-23T01:09:03.807 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.804+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff64804c1b0 con 0x7ff6480b2af0 2023-06-23T01:09:03.808 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.805+0000 7ff62f7fe700 1 -- 172.21.15.39:0/3974019488 <== mon.0 v2:172.21.15.39:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16927 (secure 0 0 0) 0x7ff63c054fa0 con 0x7ff6480b2af0 2023-06-23T01:09:03.808 INFO:teuthology.orchestra.run.smithi039.stdout: 2023-06-23T01:09:03.808 INFO:teuthology.orchestra.run.smithi039.stdout:{"epoch":22,"fsid":"314a4b16-8059-4f2c-986c-1624c6de1afb","created":"2023-06-23T01:08:37.443723+0000","modified":"2023-06-23T01:09:03.103682+0000","last_up_change":"2023-06-23T01:08:48.497542+0000","last_in_change":"2023-06-23T01:08:40.002994+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:08:49.048645+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:08:55.930040+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:09:00.734607+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T01:09:02.080383+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a33e557d-f70e-4a1c-a10e-bc7f351dee53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6816","nonce":117964},{"type":"v1","addr":"172.21.15.39:6817","nonce":117964}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6818","nonce":117964},{"type":"v1","addr":"172.21.15.39:6819","nonce":117964}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6822","nonce":117964},{"type":"v1","addr":"172.21.15.39:6823","nonce":117964}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6820","nonce":117964},{"type":"v1","addr":"172.21.15.39:6821","nonce":117964}]},"public_addr":"172.21.15.39:6817/117964","cluster_addr":"172.21.15.39:6819/117964","heartbeat_back_addr":"172.21.15.39:6823/117964","heartbeat_front_addr":"172.21.15.39:6821/117964","state":["exists","up"]},{"osd":1,"uuid":"757696fc-3c60-4f7f-822d-a8ff1e89534d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6800","nonce":117963},{"type":"v1","addr":"172.21.15.39:6801","nonce":117963}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6802","nonce":117963},{"type":"v1","addr":"172.21.15.39:6803","nonce":117963}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6806","nonce":117963},{"type":"v1","addr":"172.21.15.39:6807","nonce":117963}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6804","nonce":117963},{"type":"v1","addr":"172.21.15.39:6805","nonce":117963}]},"public_addr":"172.21.15.39:6801/117963","cluster_addr":"172.21.15.39:6803/117963","heartbeat_back_addr":"172.21.15.39:6807/117963","heartbeat_front_addr":"172.21.15.39:6805/117963","state":["exists","up"]},{"osd":2,"uuid":"cffcd6d2-b414-40b0-8967-26421b058b3e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6824","nonce":117974},{"type":"v1","addr":"172.21.15.39:6825","nonce":117974}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6826","nonce":117974},{"type":"v1","addr":"172.21.15.39:6827","nonce":117974}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6830","nonce":117974},{"type":"v1","addr":"172.21.15.39:6831","nonce":117974}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6828","nonce":117974},{"type":"v1","addr":"172.21.15.39:6829","nonce":117974}]},"public_addr":"172.21.15.39:6825/117974","cluster_addr":"172.21.15.39:6827/117974","heartbeat_back_addr":"172.21.15.39:6831/117974","heartbeat_front_addr":"172.21.15.39:6829/117974","state":["exists","up"]},{"osd":3,"uuid":"5a91897e-5924-4d57-ac5e-bfd06cf4ff93","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6808","nonce":117973},{"type":"v1","addr":"172.21.15.39:6809","nonce":117973}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6810","nonce":117973},{"type":"v1","addr":"172.21.15.39:6811","nonce":117973}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6814","nonce":117973},{"type":"v1","addr":"172.21.15.39:6815","nonce":117973}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.39:6812","nonce":117973},{"type":"v1","addr":"172.21.15.39:6813","nonce":117973}]},"public_addr":"172.21.15.39:6809/117973","cluster_addr":"172.21.15.39:6811/117973","heartbeat_back_addr":"172.21.15.39:6815/117973","heartbeat_front_addr":"172.21.15.39:6813/117973","state":["exists","up"]},{"osd":4,"uuid":"67ba6409-2b6b-4ad4-98c5-cf0d23c7a1a5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":118146},{"type":"v1","addr":"172.21.15.73:6801","nonce":118146}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":118146},{"type":"v1","addr":"172.21.15.73:6803","nonce":118146}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":118146},{"type":"v1","addr":"172.21.15.73:6807","nonce":118146}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":118146},{"type":"v1","addr":"172.21.15.73:6805","nonce":118146}]},"public_addr":"172.21.15.73:6801/118146","cluster_addr":"172.21.15.73:6803/118146","heartbeat_back_addr":"172.21.15.73:6807/118146","heartbeat_front_addr":"172.21.15.73:6805/118146","state":["exists","up"]},{"osd":5,"uuid":"390058ef-ef47-475e-af72-8e2f51ee184c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":118148},{"type":"v1","addr":"172.21.15.73:6817","nonce":118148}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":118148},{"type":"v1","addr":"172.21.15.73:6819","nonce":118148}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":118148},{"type":"v1","addr":"172.21.15.73:6823","nonce":118148}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":118148},{"type":"v1","addr":"172.21.15.73:6821","nonce":118148}]},"public_addr":"172.21.15.73:6817/118148","cluster_addr":"172.21.15.73:6819/118148","heartbeat_back_addr":"172.21.15.73:6823/118148","heartbeat_front_addr":"172.21.15.73:6821/118148","state":["exists","up"]},{"osd":6,"uuid":"c34ad563-5018-4cf4-830f-2ac56a3a5e5f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":118147},{"type":"v1","addr":"172.21.15.73:6809","nonce":118147}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":118147},{"type":"v1","addr":"172.21.15.73:6811","nonce":118147}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":118147},{"type":"v1","addr":"172.21.15.73:6815","nonce":118147}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":118147},{"type":"v1","addr":"172.21.15.73:6813","nonce":118147}]},"public_addr":"172.21.15.73:6809/118147","cluster_addr":"172.21.15.73:6811/118147","heartbeat_back_addr":"172.21.15.73:6815/118147","heartbeat_front_addr":"172.21.15.73:6813/118147","state":["exists","up"]},{"osd":7,"uuid":"e2023d49-2e44-412c-8cd1-c67447567c32","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":118189},{"type":"v1","addr":"172.21.15.73:6825","nonce":118189}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":118189},{"type":"v1","addr":"172.21.15.73:6827","nonce":118189}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":118189},{"type":"v1","addr":"172.21.15.73:6831","nonce":118189}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":118189},{"type":"v1","addr":"172.21.15.73:6829","nonce":118189}]},"public_addr":"172.21.15.73:6825/118189","cluster_addr":"172.21.15.73:6827/118189","heartbeat_back_addr":"172.21.15.73:6831/118189","heartbeat_front_addr":"172.21.15.73:6829/118189","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.198991+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.140721+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.167693+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.101533+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.095800+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.189388+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.133372+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:08:45.190656+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:09:03.810 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff634066e90 msgr2=0x7ff634069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.810 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff634066e90 0x7ff634069340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff6480caa10 tx=0x7ff638017000 comp rx=0 tx=0).stop 2023-06-23T01:09:03.811 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2af0 msgr2=0x7ff6480c7a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.811 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2af0 0x7ff6480c7a30 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff63c00ec80 tx=0x7ff63c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.811 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 shutdown_connections 2023-06-23T01:09:03.811 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff634066e90 0x7ff634069340 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.811 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff64815af60 0x7ff6480c9520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.811 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff6480b2720 0x7ff6480c74f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.811 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 --2- 172.21.15.39:0/3974019488 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff6480b2af0 0x7ff6480c7a30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.812 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 >> 172.21.15.39:0/3974019488 conn(0x7ff648067450 msgr2=0x7ff648158cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:03.812 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 shutdown_connections 2023-06-23T01:09:03.812 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.807+0000 7ff64dbbc700 1 -- 172.21.15.39:0/3974019488 wait complete. 2023-06-23T01:09:03.823 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-23T01:09:03.943 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.940+0000 7ff93383b700 1 Processor -- start 2023-06-23T01:09:03.944 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.941+0000 7ff93383b700 1 -- start start 2023-06-23T01:09:03.946 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.941+0000 7ff93383b700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 0x7ff92c0cb5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.941+0000 7ff93383b700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff92c0cf210 0x7ff92c15d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.941+0000 7ff93383b700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff92c0d03e0 con 0x7ff92c15ca40 2023-06-23T01:09:03.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.941+0000 7ff93383b700 1 -- --> v2:172.21.15.39:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff92c0cc930 con 0x7ff92c0cf210 2023-06-23T01:09:03.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.941+0000 7ff93383b700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0d0f80 con 0x7ff92c15ce10 2023-06-23T01:09:03.947 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.941+0000 7ff931dd8700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff92c0cf210 0x7ff92c15d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.941+0000 7ff9315d7700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7ff92c15ca40 0x7ff92c0d0870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52656/0 (socket says 172.21.15.39:52656) 2023-06-23T01:09:03.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff931dd8700 1 --2- >> v2:172.21.15.39:3300/0 conn(0x7ff92c0cf210 0x7ff92c15d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58484/0 (socket says 172.21.15.39:58484) 2023-06-23T01:09:03.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff9315d7700 1 -- 172.21.15.39:0/2362801924 learned_addr learned my addr 172.21.15.39:0/2362801924 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:09:03.948 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff930dd6700 1 --2- 172.21.15.39:0/2362801924 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 0x7ff92c0cb5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff931dd8700 1 -- 172.21.15.39:0/2362801924 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 msgr2=0x7ff92c0cb5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:03.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff931dd8700 1 --2- 172.21.15.39:0/2362801924 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 0x7ff92c0cb5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff931dd8700 1 -- 172.21.15.39:0/2362801924 >> v1:172.21.15.39:6789/0 conn(0x7ff92c15ca40 legacy=0x7ff92c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.949 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff931dd8700 1 -- 172.21.15.39:0/2362801924 --> v2:172.21.15.39:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff92c0cf7d0 con 0x7ff92c0cf210 2023-06-23T01:09:03.950 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff92bfff700 1 -- 172.21.15.39:0/2362801924 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119886898 0 0) 0x7ff92c0d03e0 con 0x7ff92c15ca40 2023-06-23T01:09:03.950 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff930dd6700 1 --2- 172.21.15.39:0/2362801924 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 0x7ff92c0cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:09:03.950 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff931dd8700 1 --2- 172.21.15.39:0/2362801924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff92c0cf210 0x7ff92c15d350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff9240098e0 tx=0x7ff9240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f75ac8a0f4ec3166 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.950 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff92bfff700 1 -- 172.21.15.39:0/2362801924 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff924009e80 con 0x7ff92c0cf210 2023-06-23T01:09:03.950 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff92bfff700 1 -- 172.21.15.39:0/2362801924 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff92400b860 con 0x7ff92c0cf210 2023-06-23T01:09:03.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.942+0000 7ff92bfff700 1 -- 172.21.15.39:0/2362801924 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff92400bbe0 con 0x7ff92c0cf210 2023-06-23T01:09:03.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.943+0000 7ff93383b700 1 -- 172.21.15.39:0/2362801924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff92c0cf210 msgr2=0x7ff92c15d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.943+0000 7ff93383b700 1 --2- 172.21.15.39:0/2362801924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff92c0cf210 0x7ff92c15d350 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff9240098e0 tx=0x7ff9240091a0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.951 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.943+0000 7ff93383b700 1 -- 172.21.15.39:0/2362801924 shutdown_connections 2023-06-23T01:09:03.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.943+0000 7ff93383b700 1 --2- 172.21.15.39:0/2362801924 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff92c0cf210 0x7ff92c15d350 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.943+0000 7ff93383b700 1 --2- 172.21.15.39:0/2362801924 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 0x7ff92c0cb5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.943+0000 7ff93383b700 1 -- 172.21.15.39:0/2362801924 >> 172.21.15.39:0/2362801924 conn(0x7ff92c067450 msgr2=0x7ff92c067850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:03.952 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.943+0000 7ff93383b700 1 -- 172.21.15.39:0/2362801924 shutdown_connections 2023-06-23T01:09:03.953 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.943+0000 7ff93383b700 1 -- 172.21.15.39:0/2362801924 wait complete. 2023-06-23T01:09:03.956 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.944+0000 7ff93383b700 1 Processor -- start 2023-06-23T01:09:03.956 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.944+0000 7ff93383b700 1 -- start start 2023-06-23T01:09:03.956 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff93383b700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff92c0cf210 0x7ff92c0b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.957 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff93383b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff92c15ca40 0x7ff92c14ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.957 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff93383b700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 0x7ff92c169450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.957 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff93383b700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c0c7700 con 0x7ff92c0cf210 2023-06-23T01:09:03.958 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff93383b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c169fa0 con 0x7ff92c15ca40 2023-06-23T01:09:03.958 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff93383b700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff92c16a280 con 0x7ff92c15ce10 2023-06-23T01:09:03.958 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff92c0cf210 0x7ff92c0b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.958 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff92c0cf210 0x7ff92c0b4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58494/0 (socket says 172.21.15.39:58494) 2023-06-23T01:09:03.959 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 -- 172.21.15.39:0/2147055326 learned_addr learned my addr 172.21.15.39:0/2147055326 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:03.959 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff931dd8700 1 --2- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 0x7ff92c169450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.959 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff930dd6700 1 --2- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff92c15ca40 0x7ff92c14ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.960 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 -- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 msgr2=0x7ff92c169450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.960 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 --2- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7ff92c15ce10 0x7ff92c169450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.960 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 -- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff92c15ca40 msgr2=0x7ff92c14ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:03.960 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 --2- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff92c15ca40 0x7ff92c14ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 -- 172.21.15.39:0/2147055326 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff924009420 con 0x7ff92c0cf210 2023-06-23T01:09:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.945+0000 7ff9315d7700 1 --2- 172.21.15.39:0/2147055326 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7ff92c0cf210 0x7ff92c0b4400 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff91c00cb10 tx=0x7ff91c0017b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.946+0000 7ff9297fa700 1 -- 172.21.15.39:0/2147055326 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c00ce20 con 0x7ff92c0cf210 2023-06-23T01:09:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.946+0000 7ff9297fa700 1 -- 172.21.15.39:0/2147055326 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff91c005d80 con 0x7ff92c0cf210 2023-06-23T01:09:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.946+0000 7ff9297fa700 1 -- 172.21.15.39:0/2147055326 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff91c00a820 con 0x7ff92c0cf210 2023-06-23T01:09:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.946+0000 7ff93383b700 1 -- 172.21.15.39:0/2147055326 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff92c0c79a0 con 0x7ff92c0cf210 2023-06-23T01:09:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.946+0000 7ff93383b700 1 -- 172.21.15.39:0/2147055326 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff92c0ca320 con 0x7ff92c0cf210 2023-06-23T01:09:03.961 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.947+0000 7ff93383b700 1 -- 172.21.15.39:0/2147055326 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff92c04c1b0 con 0x7ff92c0cf210 2023-06-23T01:09:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.947+0000 7ff9297fa700 1 -- 172.21.15.39:0/2147055326 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7ff91c00ac10 con 0x7ff92c0cf210 2023-06-23T01:09:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.947+0000 7ff9297fa700 1 --2- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff918066e90 0x7ff918069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.947+0000 7ff9297fa700 1 -- 172.21.15.39:0/2147055326 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff91c083ea0 con 0x7ff92c0cf210 2023-06-23T01:09:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.948+0000 7ff930dd6700 1 --2- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff918066e90 0x7ff918069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.948+0000 7ff930dd6700 1 --2- 172.21.15.39:0/2147055326 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7ff918066e90 0x7ff918069340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0c9e30 tx=0x7ff920008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:03.962 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:03.952+0000 7ff9297fa700 1 -- 172.21.15.39:0/2147055326 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7ff91c0510f0 con 0x7ff92c0cf210 2023-06-23T01:09:04.196 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 1 closest matches: 2023-06-23T01:09:04.196 INFO:teuthology.orchestra.run.smithi039.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:09:04.197 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2023-06-23T01:09:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:09:04.200 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-23T01:09:04.201 DEBUG:teuthology.orchestra.run.smithi039:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-23T01:09:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.319+0000 7f3a163d8700 1 Processor -- start 2023-06-23T01:09:04.322 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.319+0000 7f3a163d8700 1 -- start start 2023-06-23T01:09:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a163d8700 1 --2- >> v2:172.21.15.73:3300/0 conn(0x7f3a10067b60 0x7f3a1015a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:04.325 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a163d8700 1 -- --> v1:172.21.15.39:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a100ce670 con 0x7f3a10156ca0 2023-06-23T01:09:04.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a163d8700 1 -- --> v1:172.21.15.73:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1015d5f0 con 0x7f3a10157070 2023-06-23T01:09:04.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a163d8700 1 -- --> v2:172.21.15.73:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1015de90 con 0x7f3a10067b60 2023-06-23T01:09:04.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a153d6700 1 --1- >> v1:172.21.15.39:6789/0 conn(0x7f3a10156ca0 0x7f3a10067450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.39:6789/0 says I am v1:172.21.15.39:52664/0 (socket says 172.21.15.39:52664) 2023-06-23T01:09:04.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a153d6700 1 -- 172.21.15.39:0/3532092148 learned_addr learned my addr 172.21.15.39:0/3532092148 (peer_addr_for_me v1:172.21.15.39:0/0) 2023-06-23T01:09:04.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a14bd5700 1 --2- 172.21.15.39:0/3532092148 >> v2:172.21.15.73:3300/0 conn(0x7f3a10067b60 0x7f3a1015a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:04.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 <== mon.0 v1:172.21.15.39:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674260281 0 0) 0x7f3a100ce670 con 0x7f3a10156ca0 2023-06-23T01:09:04.326 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.320+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 --> v1:172.21.15.39:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39fc003570 con 0x7f3a10156ca0 2023-06-23T01:09:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 <== mon.1 v1:172.21.15.73:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463491938 0 0) 0x7f3a1015d5f0 con 0x7f3a10157070 2023-06-23T01:09:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 --> v1:172.21.15.73:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a100ce670 con 0x7f3a10157070 2023-06-23T01:09:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 <== mon.0 v1:172.21.15.39:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 564205026 0 0) 0x7f39fc003570 con 0x7f3a10156ca0 2023-06-23T01:09:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 >> v2:172.21.15.73:3300/0 conn(0x7f3a10067b60 msgr2=0x7f3a1015a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 --2- 172.21.15.39:0/3532092148 >> v2:172.21.15.73:3300/0 conn(0x7f3a10067b60 0x7f3a1015a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 >> v1:172.21.15.73:6789/0 conn(0x7f3a10157070 legacy=0x7f3a100ceb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:04.327 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 --> v1:172.21.15.39:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1015d8d0 con 0x7f3a10156ca0 2023-06-23T01:09:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 <== mon.0 v1:172.21.15.39:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 979813403 0 0) 0x7f3a0c001470 con 0x7f3a10156ca0 2023-06-23T01:09:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 >> v1:172.21.15.39:6789/0 conn(0x7f3a10156ca0 legacy=0x7f3a10067450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f39fc003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f39fc004810 0x7f39fc004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39fc0051c0 0x7f39fc00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:04.328 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1015de90 con 0x7f39fc003bf0 2023-06-23T01:09:04.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc00c0b0 con 0x7f39fc0051c0 2023-06-23T01:09:04.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39fc00c270 con 0x7f39fc004810 2023-06-23T01:09:04.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a14bd5700 1 --2- 172.21.15.39:0/3532092148 >> v2:172.21.15.73:3300/0 conn(0x7f3a10067b60 0x7f3a1015a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:09:04.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a153d6700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f39fc003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:04.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a15bd7700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39fc0051c0 0x7f39fc00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:04.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a14bd5700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f39fc004810 0x7f39fc004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:04.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a153d6700 1 -- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f39fc004810 msgr2=0x7f39fc004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:04.329 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a153d6700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f39fc004810 0x7f39fc004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a153d6700 1 -- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39fc0051c0 msgr2=0x7f39fc00ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:09:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a153d6700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39fc0051c0 0x7f39fc00ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a153d6700 1 -- 172.21.15.39:0/3532092148 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc003570 con 0x7f39fc003bf0 2023-06-23T01:09:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a15bd7700 1 -- 172.21.15.39:0/3532092148 reap_dead start 2023-06-23T01:09:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a14bd5700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f39fc004810 0x7f39fc004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:09:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a153d6700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f39fc003fe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00c2d0 tx=0x7f3a0c001e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c798c7aae064de3 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:04.330 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.321+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00cba0 con 0x7f39fc003bf0 2023-06-23T01:09:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c00cd00 con 0x7f39fc003bf0 2023-06-23T01:09:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a07fff700 1 -- 172.21.15.39:0/3532092148 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c005b70 con 0x7f39fc003bf0 2023-06-23T01:09:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a163d8700 1 -- 172.21.15.39:0/3532092148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 msgr2=0x7f39fc003fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a163d8700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f39fc003fe0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00c2d0 tx=0x7f3a0c001e10 comp rx=0 tx=0).stop 2023-06-23T01:09:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a163d8700 1 -- 172.21.15.39:0/3532092148 shutdown_connections 2023-06-23T01:09:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a163d8700 1 --2- 172.21.15.39:0/3532092148 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f39fc003fe0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:04.331 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a163d8700 1 -- 172.21.15.39:0/3532092148 >> 172.21.15.39:0/3532092148 conn(0x7f3a100d0240 msgr2=0x7f3a100d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:09:04.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a163d8700 1 -- 172.21.15.39:0/3532092148 shutdown_connections 2023-06-23T01:09:04.332 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.322+0000 7f3a163d8700 1 -- 172.21.15.39:0/3532092148 wait complete. 2023-06-23T01:09:04.334 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.323+0000 7f3a163d8700 1 Processor -- start 2023-06-23T01:09:04.334 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.323+0000 7f3a163d8700 1 -- start start 2023-06-23T01:09:04.334 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a163d8700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f3a10146ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:04.335 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a163d8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a100b0ed0 0x7f3a100b1340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:04.335 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a163d8700 1 --2- >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3a100c95f0 0x7f3a100c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:04.335 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a163d8700 1 -- --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b1880 con 0x7f39fc003bf0 2023-06-23T01:09:04.335 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a163d8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b3440 con 0x7f3a100b0ed0 2023-06-23T01:09:04.335 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a163d8700 1 -- --> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a100b3110 con 0x7f3a100c95f0 2023-06-23T01:09:04.335 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a153d6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f3a10146ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:04.335 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a153d6700 1 --2- >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f3a10146ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.39:3300/0 says I am v2:172.21.15.39:58506/0 (socket says 172.21.15.39:58506) 2023-06-23T01:09:04.335 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a153d6700 1 -- 172.21.15.39:0/3397250186 learned_addr learned my addr 172.21.15.39:0/3397250186 (peer_addr_for_me v2:172.21.15.39:0/0) 2023-06-23T01:09:04.336 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a14bd5700 1 --2- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a100b0ed0 0x7f3a100b1340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:04.336 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.324+0000 7f3a15bd7700 1 --2- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3a100c95f0 0x7f3a100c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:04.336 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a153d6700 1 -- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3a100c95f0 msgr2=0x7f3a100c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:04.336 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a153d6700 1 --2- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:3301/0,v1:172.21.15.73:6790/0] conn(0x7f3a100c95f0 0x7f3a100c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:04.336 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a153d6700 1 -- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a100b0ed0 msgr2=0x7f3a100b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:09:04.336 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a153d6700 1 --2- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3a100b0ed0 0x7f3a100b1340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:09:04.336 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a153d6700 1 -- 172.21.15.39:0/3397250186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a100ca9a0 con 0x7f39fc003bf0 2023-06-23T01:09:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a153d6700 1 --2- 172.21.15.39:0/3397250186 >> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] conn(0x7f39fc003bf0 0x7f3a10146ba0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c007c90 tx=0x7f3a0c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a057fa700 1 -- 172.21.15.39:0/3397250186 <== mon.0 v2:172.21.15.39:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c004ac0 con 0x7f39fc003bf0 2023-06-23T01:09:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a057fa700 1 -- 172.21.15.39:0/3397250186 <== mon.0 v2:172.21.15.39:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a0c004c20 con 0x7f39fc003bf0 2023-06-23T01:09:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a163d8700 1 -- 172.21.15.39:0/3397250186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a100ca3e0 con 0x7f39fc003bf0 2023-06-23T01:09:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a057fa700 1 -- 172.21.15.39:0/3397250186 <== mon.0 v2:172.21.15.39:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c0070d0 con 0x7f39fc003bf0 2023-06-23T01:09:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.325+0000 7f3a163d8700 1 -- 172.21.15.39:0/3397250186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a101470e0 con 0x7f39fc003bf0 2023-06-23T01:09:04.337 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.326+0000 7f3a163d8700 1 -- 172.21.15.39:0/3397250186 --> [v2:172.21.15.39:3300/0,v1:172.21.15.39:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a100cab30 con 0x7f39fc003bf0 2023-06-23T01:09:04.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.326+0000 7f3a057fa700 1 -- 172.21.15.39:0/3397250186 <== mon.0 v2:172.21.15.39:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f3a0c007460 con 0x7f39fc003bf0 2023-06-23T01:09:04.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.327+0000 7f3a057fa700 1 --2- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f39f4066e90 0x7f39f4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:09:04.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.327+0000 7f3a057fa700 1 -- 172.21.15.39:0/3397250186 <== mon.0 v2:172.21.15.39:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f3a0c085220 con 0x7f39fc003bf0 2023-06-23T01:09:04.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.327+0000 7f3a14bd5700 1 --2- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f39f4066e90 0x7f39f4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:09:04.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.327+0000 7f3a14bd5700 1 --2- 172.21.15.39:0/3397250186 >> [v2:172.21.15.73:6832/117558,v1:172.21.15.73:6833/117558] conn(0x7f39f4066e90 0x7f39f4069340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3a000093f0 tx=0x7f3a00008040 comp rx=0 tx=0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:09:04.338 INFO:teuthology.orchestra.run.smithi039.stderr:2023-06-23T01:09:04.331+0000 7f3a057fa700 1 -- 172.21.15.39:0/3397250186 <== mon.0 v2:172.21.15.39:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f3a0c0523f0 con 0x7f39fc003bf0 2023-06-23T01:09:04.580 INFO:teuthology.orchestra.run.smithi039.stderr:no valid command found; 1 closest matches: 2023-06-23T01:09:04.580 INFO:teuthology.orchestra.run.smithi039.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:09:04.580 INFO:teuthology.orchestra.run.smithi039.stderr:Error EINVAL: invalid command 2023-06-23T01:09:04.583 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:09:04.583 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:09:04.584 INFO:teuthology.misc:Shutting down mds daemons... 2023-06-23T01:09:04.585 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-23T01:09:04.585 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.592 INFO:tasks.ceph.mds.a:Stopped 2023-06-23T01:09:04.593 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-23T01:09:04.593 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.642 INFO:tasks.ceph.mds.c:Stopped 2023-06-23T01:09:04.643 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-23T01:09:04.643 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.655 INFO:tasks.ceph.mds.b:Stopped 2023-06-23T01:09:04.656 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-23T01:09:04.656 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.706 INFO:tasks.ceph.mds.d:Stopped 2023-06-23T01:09:04.706 INFO:teuthology.misc:Shutting down osd daemons... 2023-06-23T01:09:04.707 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-06-23T01:09:04.707 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.737 INFO:tasks.ceph.osd.0:Stopped 2023-06-23T01:09:04.738 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-06-23T01:09:04.739 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.812 INFO:tasks.ceph.osd.1:Stopped 2023-06-23T01:09:04.813 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-06-23T01:09:04.814 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.880 INFO:tasks.ceph.osd.2:Stopped 2023-06-23T01:09:04.880 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-06-23T01:09:04.881 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.948 INFO:tasks.ceph.osd.3:Stopped 2023-06-23T01:09:04.949 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-06-23T01:09:04.950 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:04.985 INFO:tasks.ceph.osd.4:Stopped 2023-06-23T01:09:04.986 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-06-23T01:09:04.986 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:05.061 INFO:tasks.ceph.osd.5:Stopped 2023-06-23T01:09:05.062 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-06-23T01:09:05.062 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:05.132 INFO:tasks.ceph.osd.6:Stopped 2023-06-23T01:09:05.132 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-06-23T01:09:05.133 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:05.203 INFO:tasks.ceph.osd.7:Stopped 2023-06-23T01:09:05.204 INFO:teuthology.misc:Shutting down mgr daemons... 2023-06-23T01:09:05.205 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-06-23T01:09:05.206 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:05.228 INFO:tasks.ceph.mgr.y:Stopped 2023-06-23T01:09:05.229 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-06-23T01:09:05.229 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:05.269 INFO:tasks.ceph.mgr.x:Stopped 2023-06-23T01:09:05.270 INFO:teuthology.misc:Shutting down mon daemons... 2023-06-23T01:09:05.270 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-06-23T01:09:05.271 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:05.280 INFO:tasks.ceph.mon.a:Stopped 2023-06-23T01:09:05.281 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-06-23T01:09:05.282 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:05.320 INFO:tasks.ceph.mon.b:Stopped 2023-06-23T01:09:05.321 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-06-23T01:09:05.322 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:09:05.371 INFO:tasks.ceph.mon.c:Stopped 2023-06-23T01:09:05.372 INFO:tasks.ceph:Checking cluster log for badness... 2023-06-23T01:09:05.372 DEBUG:teuthology.orchestra.run.smithi039:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'bad backtrace' | egrep -v 'object missing on disk' | egrep -v 'error reading table object' | egrep -v 'error reading sessionmap' | egrep -v 'unmatched fragstat' | egrep -v 'unmatched rstat' | egrep -v 'was unreadable, recreating it now' | egrep -v 'Scrub error on inode' | egrep -v 'Metadata damage detected' | egrep -v 'inconsistent rstat on inode' | egrep -v 'Error recovering journal' | head -n 1 2023-06-23T01:09:05.404 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:09:05.405 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-06-23T01:09:05.728 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:09:05.729 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-06-23T01:09:05.846 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:09:05.847 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-06-23T01:09:05.908 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:09:05.909 DEBUG:teuthology.orchestra.run.smithi039:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-06-23T01:09:05.969 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:09:05.970 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-06-23T01:09:06.206 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:09:06.206 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-06-23T01:09:06.281 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:09:06.281 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-06-23T01:09:06.354 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:09:06.355 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-06-23T01:09:06.439 INFO:tasks.ceph:Archiving mon data... 2023-06-23T01:09:06.441 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/data/mon.a.tgz 2023-06-23T01:09:06.442 DEBUG:teuthology.orchestra.run.smithi039:> mktemp 2023-06-23T01:09:06.457 INFO:teuthology.orchestra.run.smithi039.stdout:/tmp/tmp.IBuACBjxn6 2023-06-23T01:09:06.458 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.IBuACBjxn6 2023-06-23T01:09:06.571 DEBUG:teuthology.orchestra.run.smithi039:> sudo chmod 0666 /tmp/tmp.IBuACBjxn6 2023-06-23T01:09:06.656 DEBUG:teuthology.orchestra.remote:smithi039:/tmp/tmp.IBuACBjxn6 is 199KB 2023-06-23T01:09:06.709 DEBUG:teuthology.orchestra.run.smithi039:> rm -fr /tmp/tmp.IBuACBjxn6 2023-06-23T01:09:06.723 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/data/mon.b.tgz 2023-06-23T01:09:06.723 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2023-06-23T01:09:06.739 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.2TpkRrRuGI 2023-06-23T01:09:06.740 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.2TpkRrRuGI 2023-06-23T01:09:06.864 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.2TpkRrRuGI 2023-06-23T01:09:06.945 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.2TpkRrRuGI is 278KB 2023-06-23T01:09:07.037 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.2TpkRrRuGI 2023-06-23T01:09:07.053 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/data/mon.c.tgz 2023-06-23T01:09:07.053 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2023-06-23T01:09:07.105 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.A7PSnHAXIt 2023-06-23T01:09:07.105 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.A7PSnHAXIt 2023-06-23T01:09:07.213 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.A7PSnHAXIt 2023-06-23T01:09:07.294 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.A7PSnHAXIt is 278KB 2023-06-23T01:09:07.387 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.A7PSnHAXIt 2023-06-23T01:09:07.401 INFO:tasks.ceph:Cleaning ceph cluster... 2023-06-23T01:09:07.402 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T01:09:07.404 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T01:09:07.465 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-06-23T01:09:07.466 DEBUG:teuthology.orchestra.run.smithi039:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T01:09:07.471 DEBUG:teuthology.orchestra.run.smithi073:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T01:09:07.495 INFO:teuthology.orchestra.run.smithi039.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T01:09:07.529 INFO:teuthology.orchestra.run.smithi073.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T01:09:07.535 INFO:tasks.ceph:Archiving crash dumps... 2023-06-23T01:09:07.537 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/remote/smithi039/crash 2023-06-23T01:09:07.538 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T01:09:07.567 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/remote/smithi073/crash 2023-06-23T01:09:07.568 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T01:09:07.598 INFO:tasks.ceph:Shutting down logrotate 2023-06-23T01:09:07.599 DEBUG:teuthology.orchestra.run.smithi039:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:09:07.612 DEBUG:teuthology.orchestra.run.smithi073:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:09:07.693 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T01:09:07.718 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T01:09:07.743 INFO:tasks.ceph:Compressing logs... 2023-06-23T01:09:07.743 DEBUG:teuthology.orchestra.run.smithi039:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:09:07.761 DEBUG:teuthology.orchestra.run.smithi073:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:09:08.052 INFO:tasks.ceph:Archiving logs... 2023-06-23T01:09:08.053 DEBUG:teuthology.misc:Transferring archived files from smithi039:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/remote/smithi039/log 2023-06-23T01:09:08.058 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:09:08.125 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/remote/smithi073/log 2023-06-23T01:09:08.127 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:09:08.192 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:09:08.269 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9a1c5b8dbb43457c98d0cdda85fc4ea4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi039 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:09:08.275 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-06-23T01:09:08.293 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-23T01:09:08.302 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-23T01:09:08.303 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:09:08.334 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:09:08.412 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-23T01:09:08.424 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-23T01:09:08.424 DEBUG:teuthology.orchestra.run.smithi039:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:09:08.427 DEBUG:teuthology.orchestra.run.smithi073:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:09:08.438 INFO:teuthology.orchestra.run.smithi039.stderr:bash: ntpq: command not found 2023-06-23T01:09:08.440 INFO:teuthology.orchestra.run.smithi073.stderr:bash: ntpq: command not found 2023-06-23T01:09:08.442 INFO:teuthology.orchestra.run.smithi039.stdout:210 Number of sources = 4 2023-06-23T01:09:08.443 INFO:teuthology.orchestra.run.smithi039.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:09:08.443 INFO:teuthology.orchestra.run.smithi039.stdout:=============================================================================== 2023-06-23T01:09:08.443 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv01.front.sepia.ceph.com 2 6 77 62 +2077us[+2046us] +/- 85ms 2023-06-23T01:09:08.443 INFO:teuthology.orchestra.run.smithi039.stdout:^* hv02.front.sepia.ceph.com 3 6 77 62 -1892us[-1923us] +/- 78ms 2023-06-23T01:09:08.444 INFO:teuthology.orchestra.run.smithi039.stdout:^+ hv03.front.sepia.ceph.com 3 6 77 62 -9341ns[ -41us] +/- 84ms 2023-06-23T01:09:08.444 INFO:teuthology.orchestra.run.smithi039.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:08.445 INFO:teuthology.orchestra.run.smithi073.stdout:210 Number of sources = 4 2023-06-23T01:09:08.445 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:09:08.446 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2023-06-23T01:09:08.446 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv01.front.sepia.ceph.com 2 6 77 61 +2099us[+2099us] +/- 85ms 2023-06-23T01:09:08.446 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv02.front.sepia.ceph.com 3 6 77 61 -1903us[-1945us] +/- 78ms 2023-06-23T01:09:08.446 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv03.front.sepia.ceph.com 3 6 77 60 -12us[ -12us] +/- 84ms 2023-06-23T01:09:08.447 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:08.447 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-23T01:09:08.456 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-23T01:09:08.457 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-23T01:09:08.466 DEBUG:teuthology.orchestra.run.smithi039:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:09:08.571 DEBUG:teuthology.orchestra.run.smithi073:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:09:08.657 DEBUG:teuthology.orchestra.run.smithi039:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:09:08.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:09:08.688 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:09:08.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:09:08.718 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-23T01:09:08.731 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-23T01:09:08.741 INFO:teuthology.task.internal:Duration was 1081.113690 seconds 2023-06-23T01:09:08.741 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-23T01:09:08.751 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-23T01:09:08.751 DEBUG:teuthology.orchestra.run.smithi039:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:09:08.754 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:09:08.797 INFO:teuthology.orchestra.run.smithi039.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:09:08.801 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:09:09.129 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-23T01:09:09.130 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi039.front.sepia.ceph.com 2023-06-23T01:09:09.131 DEBUG:teuthology.orchestra.run.smithi039:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:09:09.151 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi073.front.sepia.ceph.com 2023-06-23T01:09:09.152 DEBUG:teuthology.orchestra.run.smithi073:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:09:09.171 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-23T01:09:09.171 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:09:09.194 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:09:09.248 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-23T01:09:09.249 DEBUG:teuthology.orchestra.run.smithi039:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:09:09.273 DEBUG:teuthology.orchestra.run.smithi073:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:09:09.357 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-23T01:09:09.367 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-23T01:09:09.368 DEBUG:teuthology.orchestra.run.smithi039:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:09:09.393 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:09:09.418 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-23T01:09:09.427 DEBUG:teuthology.orchestra.run.smithi039:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:09:09.435 DEBUG:teuthology.orchestra.run.smithi073:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:09:09.458 INFO:teuthology.orchestra.run.smithi039.stdout:kernel.core_pattern = core 2023-06-23T01:09:09.481 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = core 2023-06-23T01:09:09.495 DEBUG:teuthology.orchestra.run.smithi039:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:09:09.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:09:09.527 DEBUG:teuthology.orchestra.run.smithi073:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:09:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:09:09.549 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-23T01:09:09.562 INFO:teuthology.task.internal:Transferring archived files... 2023-06-23T01:09:09.563 DEBUG:teuthology.misc:Transferring archived files from smithi039:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/remote/smithi039 2023-06-23T01:09:09.564 DEBUG:teuthology.orchestra.run.smithi039:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:09:09.601 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920/remote/smithi073 2023-06-23T01:09:09.602 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:09:09.640 INFO:teuthology.task.internal:Removing archive directory... 2023-06-23T01:09:09.716 DEBUG:teuthology.orchestra.run.smithi039:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:09:09.718 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:09:09.733 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-23T01:09:09.743 INFO:teuthology.task.internal:Not uploading archives. 2023-06-23T01:09:09.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-23T01:09:09.752 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-23T01:09:09.753 DEBUG:teuthology.orchestra.run.smithi039:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:09:09.773 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:09:09.785 INFO:teuthology.orchestra.run.smithi039.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:09 /home/ubuntu/cephtest 2023-06-23T01:09:09.787 INFO:teuthology.orchestra.run.smithi073.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:09 /home/ubuntu/cephtest 2023-06-23T01:09:09.788 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-23T01:09:09.798 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-23T01:09:09.817 INFO:teuthology.nuke:Checking targets against current locks 2023-06-23T01:09:09.847 DEBUG:teuthology.nuke:shortname: smithi039 2023-06-23T01:09:09.847 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:09:09.871 DEBUG:teuthology.nuke:shortname: smithi073 2023-06-23T01:09:09.871 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:09:09.874 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi039.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:41:46.876679', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBA+FEBb0lM2JOjErPpPnTaufA95sbkevpUNdZRQxeWME9cdgBhMzKCyPEvUN+ik/0QBSqca9d17MEbfEIIxoqNw='} 2023-06-23T01:09:09.901 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311920', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:41:46.878554', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCiGlkx9VorGvQLNcRASuoLUUj4Sv96vdo3OTdVFt9DQkbdF27RlK8Tj2JUas2bUAA8yZ89j7Io0YejSM2nYka4iKTaGqgBVxa4tHUZ+d4PT8GrVn3U/U91xQCGU2+8rgcNAWMPYAC+e4xor6VjopW1+k6DKiC2tc0pL+tMD5EpPA1VVUciLB6I3gSD0Y9Dcjf2mNIzaxnDl+CW6oUcyfq/TGw6tsluKyr14CE6ZELMn1kvUM/zH/P5uQKUiQyhYKzy5JnUzlawpMUQvpCvN1AlqlY1lVyPEogq6bCRl7Rkn+61H4uA85igg/9l29bMuHi4uVHnB+Q02Em6+aVr4/PUnOOyOdvQopTeZm/ZBHYnP5gSo3I+Kk6gr+VGT+lG7F8lBaqzvCCBOWQzLXyuEAuZRwhM5+S2MNEbU5xXPMr3XGp3GrPvCZbzZDmJMqypLlRB7f26kRXilC26fl5ODcmPEK3ecNT+JMwjUxGJCmWiz/BYXrPBZCR4LdJl0OqDqEM='} 2023-06-23T01:09:09.931 INFO:teuthology.orchestra.console:Power off smithi039 2023-06-23T01:09:09.933 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:09:09.957 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:09:09.979 INFO:teuthology.orchestra.console:Power off smithi073 2023-06-23T01:09:09.981 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:09:10.007 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:09:13.994 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:09:14.034 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:09:18.112 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi039.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:09:18.146 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:09:18.326 INFO:teuthology.orchestra.console:Power off for smithi039 completed 2023-06-23T01:09:22.264 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:09:22.475 INFO:teuthology.orchestra.console:Power off for smithi073 completed 2023-06-23T01:09:22.582 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v2} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/data-scan} duration: 1081.1136903762817 failure_reason: 'Command failed on smithi039 with status 22: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=9a1c5b8dbb43457c98d0cdda85fc4ea4 status: fail success: false 2023-06-23T01:09:22.583 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T01:09:22.676 INFO:teuthology.run:FAIL