2023-06-23T00:58:58.619 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-23T00:58:58.620 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939 --description fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} --verbose -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/orig.config.yaml 2023-06-23T00:58:58.679 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T00:58:58.783 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939 branch: wip-yuri4-testing-2023-06-21-0846-quincy description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} email: yweinste@redhat.com first_in_suite: false job_id: '7311939' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-06-21-0846-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-06-21-0846-quincy sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9037 sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2023-06-21-0846-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c targets: smithi057.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmFKNFbg8LfVlJA1uLOjbxgiw4+0fk092Q7BVjQpvTvl6JuPfSD6R0l/eOW+NQaS56gZ/rfd94TfJ8AEa86TmkfMTja/kU9VUsjKaHQuZUxAD27+I+p7anHddByJwo4BpNBmGMdqcJjJRmsBw/m81tdx0lIvubMm/9WxPGJuAMbYrWkiKvLuSEKIgPUwPJOJDgq3+XIpD2Fbm4HWnzq/Lc546n+53S1O+yBK6o9Mg4mgcQoC0ePsEG03+J43LEuie2P5SG3d6smDBaCzuHFrBt1hK9Qg2OA/SoXDus/pnea8JslBKWziXKy5pN1oBfSssmfjVGHN4XYT/HRFprap/IjmCcYBqZUxFX+MrNF9iEqSpuBbQejFhIwZDHjP1bi7Jjt3pJi0xhpX6e212ovUA6fPqhH18m7hNY+iVuYkgYO44N1qTigooLRHg62G/vfSy1vondDQxeqhkQNHMEGk44AIbrEEDa/spGSpfJpZmNBdrmkuAvsuZa7PKh26zZ+k= smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOO/uyCNxlFTHtgGYzIK80IutIj5fa7IQdy3A5VOOeCfVDWpZeDUdsZg/EgE6A5K+LT9cJR/J03iCupr5cnYRzE= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestSubvolumeGroupSnapshots - tasks.cephfs.test_volumes.TestSubvolumeSnapshots teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-22_15:25:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-23T00:58:58.783 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa; will attempt to use it 2023-06-23T00:58:58.784 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks 2023-06-23T00:58:58.784 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-23T00:58:58.786 INFO:teuthology.task.internal:Checking packages... 2023-06-23T00:58:58.823 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '42ddc46e6e6582d8e29a5a982a81886fe028be2c' 2023-06-23T00:58:58.824 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-23T00:58:58.825 INFO:teuthology.packaging:ref: None 2023-06-23T00:58:58.825 INFO:teuthology.packaging:tag: None 2023-06-23T00:58:58.826 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T00:58:58.826 INFO:teuthology.packaging:sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T00:58:58.827 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T00:58:58.965 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-719.g42ddc46e 2023-06-23T00:58:58.966 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-23T00:58:58.973 INFO:teuthology.task.internal:no buildpackages task found 2023-06-23T00:58:58.973 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-23T00:58:59.021 INFO:teuthology.task.internal:Saving configuration 2023-06-23T00:58:59.044 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-23T00:58:59.055 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T00:58:59.085 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:53:57.066027', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPXnVHLcVHiZV+u9IOXBSK+o6BvLLqCq4j7WHhBGjx80NoGXvsUPtNIvNfthrR+Sc+PCOew8ItySaQvBmJTb4M='} 2023-06-23T00:58:59.113 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:53:57.064491', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOO/uyCNxlFTHtgGYzIK80IutIj5fa7IQdy3A5VOOeCfVDWpZeDUdsZg/EgE6A5K+LT9cJR/J03iCupr5cnYRzE='} 2023-06-23T00:58:59.113 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-23T00:58:59.122 INFO:teuthology.task.internal:roles: ubuntu@smithi057.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-06-23T00:58:59.122 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-06-23T00:58:59.122 INFO:teuthology.run_tasks:Running task console_log... 2023-06-23T00:58:59.242 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc1a8a0a040>, signals=[15]) 2023-06-23T00:58:59.243 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-23T00:58:59.254 INFO:teuthology.task.internal:Opening connections... 2023-06-23T00:58:59.255 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T00:58:59.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:58:59.340 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T00:58:59.341 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T00:58:59.410 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-23T00:58:59.417 DEBUG:teuthology.orchestra.run.smithi057:> uname -m 2023-06-23T00:58:59.441 INFO:teuthology.orchestra.run.smithi057.stdout:x86_64 2023-06-23T00:58:59.442 DEBUG:teuthology.orchestra.run.smithi057:> cat /etc/os-release 2023-06-23T00:58:59.503 INFO:teuthology.orchestra.run.smithi057.stdout:NAME="CentOS Stream" 2023-06-23T00:58:59.504 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION="8" 2023-06-23T00:58:59.504 INFO:teuthology.orchestra.run.smithi057.stdout:ID="centos" 2023-06-23T00:58:59.504 INFO:teuthology.orchestra.run.smithi057.stdout:ID_LIKE="rhel fedora" 2023-06-23T00:58:59.504 INFO:teuthology.orchestra.run.smithi057.stdout:VERSION_ID="8" 2023-06-23T00:58:59.504 INFO:teuthology.orchestra.run.smithi057.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:58:59.504 INFO:teuthology.orchestra.run.smithi057.stdout:PRETTY_NAME="CentOS Stream 8" 2023-06-23T00:58:59.505 INFO:teuthology.orchestra.run.smithi057.stdout:ANSI_COLOR="0;31" 2023-06-23T00:58:59.505 INFO:teuthology.orchestra.run.smithi057.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-06-23T00:58:59.505 INFO:teuthology.orchestra.run.smithi057.stdout:HOME_URL="https://centos.org/" 2023-06-23T00:58:59.505 INFO:teuthology.orchestra.run.smithi057.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:58:59.505 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:58:59.505 INFO:teuthology.orchestra.run.smithi057.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-06-23T00:58:59.507 INFO:teuthology.lock.ops:Updating smithi057.front.sepia.ceph.com on lock server 2023-06-23T00:58:59.531 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2023-06-23T00:58:59.554 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2023-06-23T00:58:59.555 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2023-06-23T00:58:59.616 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2023-06-23T00:58:59.616 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8" 2023-06-23T00:58:59.616 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2023-06-23T00:58:59.616 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2023-06-23T00:58:59.617 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8" 2023-06-23T00:58:59.617 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2023-06-23T00:58:59.617 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 8" 2023-06-23T00:58:59.617 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2023-06-23T00:58:59.617 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-06-23T00:58:59.617 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2023-06-23T00:58:59.618 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T00:58:59.618 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T00:58:59.618 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-06-23T00:58:59.619 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2023-06-23T00:58:59.648 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-23T00:58:59.657 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-23T00:58:59.696 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-23T00:58:59.697 DEBUG:teuthology.orchestra.run.smithi057:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:58:59.699 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2023-06-23T00:58:59.720 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-23T00:58:59.730 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-23T00:58:59.730 DEBUG:teuthology.orchestra.run.smithi057:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:58:59.762 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2023-06-23T00:58:59.803 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-23T00:58:59.866 INFO:teuthology.run_tasks:Running task kernel... 2023-06-23T00:58:59.887 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-06-23T00:58:59.888 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-06-23T00:58:59.889 DEBUG:teuthology.orchestra.run.smithi057:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:58:59.890 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T00:58:59.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:58:59.909 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-06-23T00:58:59.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T00:58:59.911 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-06-23T00:58:59.969 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-489.el8.x86_64 2023-06-23T00:58:59.969 INFO:teuthology.task.kernel:Running kernel on smithi123: 4.18.0-489.el8.x86_64 2023-06-23T00:58:59.969 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2023-06-23T00:58:59.974 INFO:teuthology.orchestra.run.smithi057.stdout:4.18.0-489.el8.x86_64 2023-06-23T00:58:59.974 INFO:teuthology.task.kernel:Running kernel on smithi057: 4.18.0-489.el8.x86_64 2023-06-23T00:58:59.974 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum install -y kernel 2023-06-23T00:59:08.070 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - AppStream 5.3 MB/s | 30 MB 00:05 2023-06-23T00:59:10.302 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 3.8 MB/s | 30 MB 00:07 2023-06-23T00:59:25.303 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - BaseOS 3.4 MB/s | 39 MB 00:11 2023-06-23T00:59:26.842 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 3.5 MB/s | 39 MB 00:11 2023-06-23T00:59:31.627 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras 55 kB/s | 18 kB 00:00 2023-06-23T00:59:32.036 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras common packages 28 kB/s | 6.6 kB 00:00 2023-06-23T00:59:32.925 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 47 kB/s | 18 kB 00:00 2023-06-23T00:59:33.343 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 26 kB/s | 6.6 kB 00:00 2023-06-23T00:59:34.680 INFO:teuthology.orchestra.run.smithi057.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T00:59:35.378 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 1.8 kB/s | 3.5 kB 00:01 2023-06-23T00:59:36.896 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 11 MB/s | 16 MB 00:01 2023-06-23T00:59:38.282 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 16 MB 00:03 2023-06-23T00:59:40.670 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 437 kB/s | 24 kB 00:00 2023-06-23T00:59:42.204 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 406 kB/s | 24 kB 00:00 2023-06-23T00:59:43.710 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-06-23T00:59:43.710 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-06-23T00:59:43.923 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-06-23T00:59:43.923 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-06-23T00:59:43.924 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-06-23T00:59:43.924 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-06-23T00:59:43.924 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-06-23T00:59:43.924 INFO:teuthology.orchestra.run.smithi123.stdout: kernel x86_64 4.18.0-497.el8 baseos 9.6 M 2023-06-23T00:59:43.925 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-06-23T00:59:43.925 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core x86_64 4.18.0-497.el8 baseos 42 M 2023-06-23T00:59:43.925 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules x86_64 4.18.0-497.el8 baseos 35 M 2023-06-23T00:59:43.925 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T00:59:43.926 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-06-23T00:59:43.926 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-06-23T00:59:43.926 INFO:teuthology.orchestra.run.smithi123.stdout:Install 3 Packages 2023-06-23T00:59:43.926 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T00:59:43.927 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 87 M 2023-06-23T00:59:43.927 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 95 M 2023-06-23T00:59:43.928 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-06-23T00:59:45.487 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-06-23T00:59:45.488 INFO:teuthology.orchestra.run.smithi057.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-06-23T00:59:45.713 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-06-23T00:59:45.713 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-06-23T00:59:45.714 INFO:teuthology.orchestra.run.smithi057.stdout: Package Architecture Version Repository Size 2023-06-23T00:59:45.714 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-06-23T00:59:45.714 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-06-23T00:59:45.714 INFO:teuthology.orchestra.run.smithi057.stdout: kernel x86_64 4.18.0-497.el8 baseos 9.6 M 2023-06-23T00:59:45.714 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-06-23T00:59:45.714 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-core x86_64 4.18.0-497.el8 baseos 42 M 2023-06-23T00:59:45.714 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules x86_64 4.18.0-497.el8 baseos 35 M 2023-06-23T00:59:45.715 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T00:59:45.715 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-06-23T00:59:45.715 INFO:teuthology.orchestra.run.smithi057.stdout:================================================================================ 2023-06-23T00:59:45.715 INFO:teuthology.orchestra.run.smithi057.stdout:Install 3 Packages 2023-06-23T00:59:45.715 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T00:59:45.716 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 87 M 2023-06-23T00:59:45.716 INFO:teuthology.orchestra.run.smithi057.stdout:Installed size: 95 M 2023-06-23T00:59:45.716 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-06-23T00:59:46.413 INFO:teuthology.orchestra.run.smithi123.stdout:(1/3): kernel-4.18.0-497.el8.x86_64.rpm 4.2 MB/s | 9.6 MB 00:02 2023-06-23T00:59:47.688 INFO:teuthology.orchestra.run.smithi123.stdout:(2/3): kernel-modules-4.18.0-497.el8.x86_64.rpm 9.8 MB/s | 35 MB 00:03 2023-06-23T00:59:48.130 INFO:teuthology.orchestra.run.smithi057.stdout:(1/3): kernel-4.18.0-497.el8.x86_64.rpm 4.1 MB/s | 9.6 MB 00:02 2023-06-23T00:59:49.555 INFO:teuthology.orchestra.run.smithi057.stdout:(2/3): kernel-core-4.18.0-497.el8.x86_64.rpm 11 MB/s | 42 MB 00:03 2023-06-23T00:59:50.001 INFO:teuthology.orchestra.run.smithi057.stdout:(3/3): kernel-modules-4.18.0-497.el8.x86_64.rpm 8.3 MB/s | 35 MB 00:04 2023-06-23T00:59:50.001 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:59:50.001 INFO:teuthology.orchestra.run.smithi057.stdout:Total 20 MB/s | 87 MB 00:04 2023-06-23T00:59:50.686 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-06-23T00:59:51.112 INFO:teuthology.orchestra.run.smithi123.stdout:(3/3): kernel-core-4.18.0-497.el8.x86_64.rpm 6.1 MB/s | 42 MB 00:06 2023-06-23T00:59:51.113 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-06-23T00:59:51.114 INFO:teuthology.orchestra.run.smithi123.stdout:Total 12 MB/s | 87 MB 00:07 2023-06-23T00:59:51.151 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-06-23T00:59:51.151 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-06-23T00:59:51.765 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-06-23T00:59:52.177 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-06-23T00:59:52.177 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-06-23T00:59:52.545 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-06-23T00:59:52.572 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-06-23T00:59:53.377 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-06-23T00:59:53.398 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-06-23T00:59:55.473 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-06-23T00:59:55.968 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-core-4.18.0-497.el8.x86_64 1/3 2023-06-23T00:59:56.117 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-06-23T00:59:56.642 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-core-4.18.0-497.el8.x86_64 1/3 2023-06-23T00:59:59.687 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-4.18.0-497.el8.x86_64 1/3 2023-06-23T01:00:00.075 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-modules-4.18.0-497.el8.x86_64 2/3 2023-06-23T01:00:00.186 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-497.el8.x86_64 1/3 2023-06-23T01:00:00.600 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-4.18.0-497.el8.x86_64 2/3 2023-06-23T01:00:03.662 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-4.18.0-497.el8.x86_64 2/3 2023-06-23T01:00:03.931 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : kernel-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:03.939 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-497.el8.x86_64 2/3 2023-06-23T01:00:04.235 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:25.802 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:25.824 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:26.160 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:26.160 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-4.18.0-497.el8.x86_64 1/3 2023-06-23T01:00:26.160 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-core-4.18.0-497.el8.x86_64 2/3 2023-06-23T01:00:26.830 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:26.830 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:00:26.830 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-06-23T01:00:26.830 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-4.18.0-497.el8.x86_64 kernel-core-4.18.0-497.el8.x86_64 2023-06-23T01:00:26.830 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-4.18.0-497.el8.x86_64 2023-06-23T01:00:26.830 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:00:26.831 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-06-23T01:00:26.950 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:00:27.163 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-497.el8.x86_64 2023-06-23T01:00:27.164 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-497.el8.x86_64 2023-06-23T01:00:27.164 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-497.el8.x86_64 2023-06-23T01:00:27.165 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T01:00:27.165 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T01:00:27.165 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-06-23T01:00:27.165 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T01:00:27.165 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:00:27.234 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-core-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:27.258 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-modules-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:27.370 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-497.el8.x86_64 2023-06-23T01:00:27.370 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-497.el8.x86_64 2023-06-23T01:00:27.371 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2023-06-23T01:00:27.404 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2023-06-23T01:00:27.404 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2023-06-23T01:00:27.404 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.02 2023-06-23T01:00:27.405 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 129.el8 2023-06-23T01:00:27.405 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2023-06-23T01:00:27.405 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-06-23T01:00:27.405 INFO:teuthology.orchestra.run.smithi123.stdout:Group : System Environment/Base 2023-06-23T01:00:27.405 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 9265586 2023-06-23T01:00:27.405 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2023-06-23T01:00:27.406 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-06-23T01:00:27.406 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-06-23T01:00:27.406 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-06-23T01:00:27.406 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : kbuilder.bsys.centos.org 2023-06-23T01:00:27.406 INFO:teuthology.orchestra.run.smithi123.stdout:Relocations : (not relocatable) 2023-06-23T01:00:27.406 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : CentOS BuildSystem 2023-06-23T01:00:27.406 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2023-06-23T01:00:27.407 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T01:00:27.407 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2023-06-23T01:00:27.407 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2023-06-23T01:00:27.407 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:00:27.407 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T01:00:27.408 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T01:00:27.408 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T01:00:27.408 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2023-06-23T01:00:27.408 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:00:27.408 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T01:00:27.410 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T01:00:27.410 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 4.18.0-497.el8.x86_64 2023-06-23T01:00:27.411 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T01:00:27.621 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: kernel-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:27.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-4.18.0-497.el8.x86_64 1/3 2023-06-23T01:00:27.622 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-core-4.18.0-497.el8.x86_64 2/3 2023-06-23T01:00:28.317 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : kernel-modules-4.18.0-497.el8.x86_64 3/3 2023-06-23T01:00:28.317 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:00:28.317 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-06-23T01:00:28.317 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-4.18.0-497.el8.x86_64 kernel-core-4.18.0-497.el8.x86_64 2023-06-23T01:00:28.317 INFO:teuthology.orchestra.run.smithi057.stdout: kernel-modules-4.18.0-497.el8.x86_64 2023-06-23T01:00:28.318 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:00:28.318 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-06-23T01:00:28.471 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:00:28.688 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-4.18.0-497.el8.x86_64 2023-06-23T01:00:28.689 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-497.el8.x86_64 2023-06-23T01:00:28.689 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-497.el8.x86_64 2023-06-23T01:00:28.689 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T01:00:28.689 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T01:00:28.690 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-06-23T01:00:28.690 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi057.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T01:00:28.690 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:00:28.904 INFO:teuthology.orchestra.run.smithi057.stdout:kernel-4.18.0-497.el8.x86_64 2023-06-23T01:00:28.904 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-497.el8.x86_64 2023-06-23T01:00:28.905 DEBUG:teuthology.orchestra.run.smithi057:> sudo rpm -qi grub2-tools 2023-06-23T01:00:28.939 INFO:teuthology.orchestra.run.smithi057.stdout:Name : grub2-tools 2023-06-23T01:00:28.939 INFO:teuthology.orchestra.run.smithi057.stdout:Epoch : 1 2023-06-23T01:00:28.940 INFO:teuthology.orchestra.run.smithi057.stdout:Version : 2.02 2023-06-23T01:00:28.940 INFO:teuthology.orchestra.run.smithi057.stdout:Release : 129.el8 2023-06-23T01:00:28.940 INFO:teuthology.orchestra.run.smithi057.stdout:Architecture: x86_64 2023-06-23T01:00:28.940 INFO:teuthology.orchestra.run.smithi057.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-06-23T01:00:28.940 INFO:teuthology.orchestra.run.smithi057.stdout:Group : System Environment/Base 2023-06-23T01:00:28.940 INFO:teuthology.orchestra.run.smithi057.stdout:Size : 9265586 2023-06-23T01:00:28.941 INFO:teuthology.orchestra.run.smithi057.stdout:License : GPLv3+ 2023-06-23T01:00:28.941 INFO:teuthology.orchestra.run.smithi057.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-06-23T01:00:28.941 INFO:teuthology.orchestra.run.smithi057.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-06-23T01:00:28.941 INFO:teuthology.orchestra.run.smithi057.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-06-23T01:00:28.941 INFO:teuthology.orchestra.run.smithi057.stdout:Build Host : kbuilder.bsys.centos.org 2023-06-23T01:00:28.941 INFO:teuthology.orchestra.run.smithi057.stdout:Relocations : (not relocatable) 2023-06-23T01:00:28.941 INFO:teuthology.orchestra.run.smithi057.stdout:Packager : CentOS BuildSystem 2023-06-23T01:00:28.941 INFO:teuthology.orchestra.run.smithi057.stdout:Vendor : CentOS 2023-06-23T01:00:28.942 INFO:teuthology.orchestra.run.smithi057.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T01:00:28.942 INFO:teuthology.orchestra.run.smithi057.stdout:Summary : Support tools for GRUB. 2023-06-23T01:00:28.942 INFO:teuthology.orchestra.run.smithi057.stdout:Description : 2023-06-23T01:00:28.942 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:00:28.942 INFO:teuthology.orchestra.run.smithi057.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T01:00:28.942 INFO:teuthology.orchestra.run.smithi057.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T01:00:28.942 INFO:teuthology.orchestra.run.smithi057.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T01:00:28.943 INFO:teuthology.orchestra.run.smithi057.stdout:hardware devices. 2023-06-23T01:00:28.943 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:00:28.943 INFO:teuthology.orchestra.run.smithi057.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T01:00:28.944 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T01:00:28.944 INFO:teuthology.task.kernel:Updating grub on smithi057 to boot 4.18.0-497.el8.x86_64 2023-06-23T01:00:28.944 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T01:00:29.207 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2023-06-23T01:00:30.340 INFO:teuthology.orchestra.run.smithi057.stderr:Generating grub configuration file ... 2023-06-23T01:00:31.475 INFO:teuthology.orchestra.run.smithi123.stderr:done 2023-06-23T01:00:31.477 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-06-23T01:00:31.522 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.7ObEnJyy1D 2023-06-23T01:00:31.522 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7ObEnJyy1D 2023-06-23T01:00:31.596 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.7ObEnJyy1D 2023-06-23T01:00:31.751 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.7ObEnJyy1D is 6KB 2023-06-23T01:00:31.800 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.7ObEnJyy1D 2023-06-23T01:00:31.819 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T01:00:31.916 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-06-23T01:00:31.916 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-06-23T01:00:31.916 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-06-23T01:00:31.917 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-497.el8.x86_64.conf 2023-06-23T01:00:31.917 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-497.el8.x86_64 2023-06-23T01:00:32.054 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2023-06-23T01:00:32.619 INFO:teuthology.orchestra.run.smithi057.stderr:done 2023-06-23T01:00:32.620 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2023-06-23T01:00:32.674 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.arvf7TeYGt 2023-06-23T01:00:32.675 DEBUG:teuthology.orchestra.run.smithi057:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.arvf7TeYGt 2023-06-23T01:00:32.708 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.arvf7TeYGt 2023-06-23T01:00:32.855 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.arvf7TeYGt is 6KB 2023-06-23T01:00:32.878 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.arvf7TeYGt 2023-06-23T01:00:32.896 DEBUG:teuthology.orchestra.run.smithi057:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T01:00:32.962 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-06-23T01:00:32.962 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-06-23T01:00:32.962 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-06-23T01:00:32.963 INFO:teuthology.orchestra.run.smithi057.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-497.el8.x86_64.conf 2023-06-23T01:00:32.963 DEBUG:teuthology.orchestra.run.smithi057:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-497.el8.x86_64 2023-06-23T01:00:33.130 DEBUG:teuthology.orchestra.run.smithi057:> sudo shutdown -r now 2023-06-23T01:01:02.084 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-06-23T01:01:02.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-06-23T01:01:02.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:01:03.133 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-06-23T01:01:03.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-06-23T01:01:03.134 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:01:20.624 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2023-06-23T01:01:21.648 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-06-23T01:01:50.652 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-06-23T01:01:50.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:01:51.651 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-06-23T01:01:51.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:02:07.057 DEBUG:teuthology.orchestra.run.smithi123:> true 2023-06-23T01:02:07.386 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-06-23T01:02:07.386 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-497.el8.x86_64"... 2023-06-23T01:02:07.387 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-06-23T01:02:07.406 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-497.el8.x86_64 2023-06-23T01:02:07.407 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-497.el8.x86_64 vs 4.18.0-497.el8.x86_64 2023-06-23T01:02:07.407 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T01:02:07.407 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-06-23T01:02:08.408 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-06-23T01:02:08.408 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-23T01:02:08.501 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2023-06-23T01:02:08.518 DEBUG:teuthology.parallel:result is None 2023-06-23T01:02:10.032 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.57 2023-06-23T01:02:40.033 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-06-23T01:02:40.034 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:02:40.403 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-06-23T01:02:40.695 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-06-23T01:02:40.696 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-497.el8.x86_64"... 2023-06-23T01:02:40.696 DEBUG:teuthology.orchestra.run.smithi057:> uname -r 2023-06-23T01:02:40.715 INFO:teuthology.orchestra.run.smithi057.stdout:4.18.0-497.el8.x86_64 2023-06-23T01:02:40.716 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-497.el8.x86_64 vs 4.18.0-497.el8.x86_64 2023-06-23T01:02:40.716 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T01:02:40.716 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-06-23T01:02:41.717 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-06-23T01:02:41.718 DEBUG:teuthology.orchestra.run.smithi057:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-06-23T01:02:41.816 INFO:teuthology.orchestra.run.smithi057.stdout:ttyS1 2023-06-23T01:02:41.829 DEBUG:teuthology.parallel:result is None 2023-06-23T01:02:41.829 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-23T01:02:41.839 INFO:teuthology.task.internal:Creating test directory... 2023-06-23T01:02:41.839 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T01:02:41.842 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T01:02:41.869 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-23T01:02:41.885 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-23T01:02:41.935 INFO:teuthology.task.internal:Creating archive directory... 2023-06-23T01:02:41.936 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T01:02:41.939 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T01:02:41.971 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-23T01:02:41.982 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-23T01:02:41.983 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T01:02:42.009 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T01:02:42.049 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:02:42.054 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:02:42.061 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:02:42.067 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:02:42.068 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-23T01:02:42.076 INFO:teuthology.task.internal:Configuring sudo... 2023-06-23T01:02:42.077 DEBUG:teuthology.orchestra.run.smithi057:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T01:02:42.106 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T01:02:42.139 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-23T01:02:42.152 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-23T01:02:42.152 DEBUG:teuthology.orchestra.run.smithi057:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T01:02:42.179 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T01:02:42.198 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:02:42.288 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:02:42.339 DEBUG:teuthology.orchestra.run.smithi057:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:02:42.428 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:02:42.470 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:02:42.471 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T01:02:42.539 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:02:42.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:02:42.648 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:02:42.741 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:02:42.785 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:02:42.786 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T01:02:42.858 DEBUG:teuthology.orchestra.run.smithi057:> sudo service rsyslog restart 2023-06-23T01:02:42.860 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2023-06-23T01:02:42.927 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:02:42.958 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:02:43.399 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-23T01:02:43.407 INFO:teuthology.task.internal:Starting timer... 2023-06-23T01:02:43.407 INFO:teuthology.run_tasks:Running task pcp... 2023-06-23T01:02:43.445 INFO:teuthology.run_tasks:Running task selinux... 2023-06-23T01:02:43.494 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-06-23T01:02:43.495 DEBUG:teuthology.orchestra.run.smithi057:> sudo service auditd rotate 2023-06-23T01:02:43.564 INFO:teuthology.orchestra.run.smithi057.stdout:Rotating logs: 2023-06-23T01:02:43.566 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2023-06-23T01:02:43.664 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2023-06-23T01:02:43.665 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-23T01:02:43.666 DEBUG:teuthology.orchestra.run.smithi057:> /usr/sbin/getenforce 2023-06-23T01:02:43.714 INFO:teuthology.orchestra.run.smithi057.stdout:Permissive 2023-06-23T01:02:43.715 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2023-06-23T01:02:43.738 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2023-06-23T01:02:43.739 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi057.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi123.front.sepia.ceph.com': 'permissive'} 2023-06-23T01:02:43.739 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:02:43.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:02:43.769 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:02:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:02:43.809 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-23T01:02:43.809 DEBUG:teuthology.orchestra.run.smithi057:> sudo /usr/sbin/setenforce permissive 2023-06-23T01:02:43.839 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2023-06-23T01:02:43.877 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-23T01:02:43.886 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-06-23T01:02:43.892 INFO:teuthology.repo_utils:Fetching main from origin 2023-06-23T01:02:44.039 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-23T01:02:44.059 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-23T01:02:44.060 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi057.front.sepia.ceph.com,smithi123.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-23T01:09:44.003 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi057.front.sepia.ceph.com'), Remote(name='ubuntu@smithi123.front.sepia.ceph.com')] 2023-06-23T01:09:44.005 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi057.front.sepia.ceph.com' 2023-06-23T01:09:44.006 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi057.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:09:44.076 DEBUG:teuthology.orchestra.run.smithi057:> true 2023-06-23T01:09:44.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-06-23T01:09:44.153 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:09:44.224 DEBUG:teuthology.orchestra.run.smithi123:> true 2023-06-23T01:09:44.298 INFO:teuthology.run_tasks:Running task clock... 2023-06-23T01:09:44.315 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-23T01:09:44.315 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T01:09:44.317 DEBUG:teuthology.orchestra.run.smithi057:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:09:44.319 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T01:09:44.320 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:09:44.354 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T01:09:44.369 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T01:09:44.373 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T01:09:44.387 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T01:09:44.450 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2023-06-23T01:09:44.461 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2023-06-23T01:09:44.475 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T01:09:44.489 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T01:09:44.561 INFO:teuthology.orchestra.run.smithi057.stderr:sudo: ntpd: command not found 2023-06-23T01:09:44.562 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2023-06-23T01:09:44.564 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:09:44.565 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2023-06-23T01:09:44.565 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:44.565 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:44.565 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:44.566 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:44.573 INFO:teuthology.orchestra.run.smithi057.stdout:506 Cannot talk to daemon 2023-06-23T01:09:44.589 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T01:09:44.604 INFO:teuthology.orchestra.run.smithi057.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T01:09:44.749 INFO:teuthology.orchestra.run.smithi057.stderr:bash: ntpq: command not found 2023-06-23T01:09:44.753 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:09:44.753 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2023-06-23T01:09:44.753 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:44.753 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:44.754 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:44.754 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:09:44.754 INFO:teuthology.run_tasks:Running task install... 2023-06-23T01:09:44.762 DEBUG:teuthology.task.install:project ceph 2023-06-23T01:09:44.762 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}} 2023-06-23T01:09:44.763 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-23T01:09:44.763 INFO:teuthology.task.install:Using flavor: default 2023-06-23T01:09:44.771 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-23T01:09:44.772 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-23T01:09:44.773 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T01:09:44.774 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:09:44.776 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T01:09:44.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:09:44.921 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T01:09:44.922 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T01:09:44.929 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T01:09:44.930 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T01:09:45.101 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T01:09:45.101 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:09:45.101 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T01:09:45.109 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T01:09:45.109 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:09:45.109 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T01:09:45.135 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T01:09:45.136 DEBUG:teuthology.orchestra.run.smithi057:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T01:09:45.143 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T01:09:45.144 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T01:09:45.210 DEBUG:teuthology.orchestra.run.smithi057:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T01:09:45.218 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T01:09:45.292 DEBUG:teuthology.orchestra.run.smithi057:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T01:09:45.301 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T01:09:45.362 INFO:teuthology.orchestra.run.smithi057.stdout:check_obsoletes = 1 2023-06-23T01:09:45.363 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum clean all 2023-06-23T01:09:45.372 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2023-06-23T01:09:45.374 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2023-06-23T01:09:45.638 INFO:teuthology.orchestra.run.smithi123.stdout:67 files removed 2023-06-23T01:09:45.644 INFO:teuthology.orchestra.run.smithi057.stdout:67 files removed 2023-06-23T01:09:45.666 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T01:09:45.672 DEBUG:teuthology.orchestra.run.smithi057:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T01:09:46.432 INFO:teuthology.orchestra.run.smithi057.stdout:ceph packages for x86_64 236 kB/s | 78 kB 00:00 2023-06-23T01:09:46.461 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 199 kB/s | 78 kB 00:00 2023-06-23T01:09:46.750 INFO:teuthology.orchestra.run.smithi057.stdout:ceph noarch packages 33 kB/s | 9.2 kB 00:00 2023-06-23T01:09:46.770 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 34 kB/s | 9.2 kB 00:00 2023-06-23T01:09:47.030 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 6.2 kB/s | 1.4 kB 00:00 2023-06-23T01:09:47.126 INFO:teuthology.orchestra.run.smithi057.stdout:ceph source packages 4.2 kB/s | 1.4 kB 00:00 2023-06-23T01:09:50.505 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 8.6 MB/s | 30 MB 00:03 2023-06-23T01:09:54.542 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - AppStream 4.0 MB/s | 30 MB 00:07 2023-06-23T01:10:08.237 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 3.1 MB/s | 39 MB 00:12 2023-06-23T01:10:13.659 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - BaseOS 2.9 MB/s | 39 MB 00:13 2023-06-23T01:10:14.253 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 50 kB/s | 18 kB 00:00 2023-06-23T01:10:14.812 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.6 kB 00:00 2023-06-23T01:10:15.689 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - PowerTools 8.2 MB/s | 6.0 MB 00:00 2023-06-23T01:10:19.657 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 1.2 kB/s | 3.5 kB 00:02 2023-06-23T01:10:20.074 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras 50 kB/s | 18 kB 00:00 2023-06-23T01:10:20.508 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - Extras common packages 25 kB/s | 6.6 kB 00:00 2023-06-23T01:10:21.293 INFO:teuthology.orchestra.run.smithi057.stdout:CentOS Stream 8 - PowerTools 9.5 MB/s | 6.0 MB 00:00 2023-06-23T01:10:21.534 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 9.0 MB/s | 16 MB 00:01 2023-06-23T01:10:24.986 INFO:teuthology.orchestra.run.smithi057.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T01:10:25.257 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 406 kB/s | 24 kB 00:00 2023-06-23T01:10:28.953 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:10:28.953 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:10:28.966 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2023-06-23T01:10:28.966 INFO:teuthology.orchestra.run.smithi123.stdout:Package NetworkManager-initscripts-updown-1:1.40.16-4.el8.noarch is already installed. 2023-06-23T01:10:28.967 INFO:teuthology.orchestra.run.smithi123.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2023-06-23T01:10:28.968 INFO:teuthology.orchestra.run.smithi123.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2023-06-23T01:10:28.969 INFO:teuthology.orchestra.run.smithi123.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T01:10:28.971 INFO:teuthology.orchestra.run.smithi123.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T01:10:28.973 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-06-23T01:10:28.974 INFO:teuthology.orchestra.run.smithi123.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-06-23T01:10:29.145 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-06-23T01:10:29.158 INFO:teuthology.orchestra.run.smithi123.stdout:==================================================================================================== 2023-06-23T01:10:29.159 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-06-23T01:10:29.159 INFO:teuthology.orchestra.run.smithi123.stdout:==================================================================================================== 2023-06-23T01:10:29.159 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-06-23T01:10:29.159 INFO:teuthology.orchestra.run.smithi123.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2023-06-23T01:10:29.159 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T01:10:29.159 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T01:10:29.160 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T01:10:29.160 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T01:10:29.160 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T01:10:29.160 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T01:10:29.160 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T01:10:29.160 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T01:10:29.160 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T01:10:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:10:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T01:10:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T01:10:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T01:10:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T01:10:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T01:10:29.161 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel x86_64 0.189-2.el8 baseos 61 k 2023-06-23T01:10:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2023-06-23T01:10:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2023-06-23T01:10:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T01:10:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T01:10:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T01:10:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2023-06-23T01:10:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel x86_64 2.32.1-42.el8 baseos 99 k 2023-06-23T01:10:29.162 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel x86_64 1:1.1.1k-9.el8 baseos 3.2 M 2023-06-23T01:10:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T01:10:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T01:10:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T01:10:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T01:10:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T01:10:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:10:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T01:10:29.163 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2023-06-23T01:10:29.164 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2023-06-23T01:10:29.164 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-7.el8 baseos 143 k 2023-06-23T01:10:29.164 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-debuginfod-client x86_64 0.189-2.el8 baseos 76 k 2023-06-23T01:10:29.164 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf x86_64 0.189-2.el8 baseos 232 k 2023-06-23T01:10:29.164 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libs x86_64 0.189-2.el8 baseos 302 k 2023-06-23T01:10:29.164 INFO:teuthology.orchestra.run.smithi123.stdout: iproute x86_64 6.2.0-2.el8 baseos 880 k 2023-06-23T01:10:29.164 INFO:teuthology.orchestra.run.smithi123.stdout: libblkid x86_64 2.32.1-42.el8 baseos 220 k 2023-06-23T01:10:29.165 INFO:teuthology.orchestra.run.smithi123.stdout: libfdisk x86_64 2.32.1-42.el8 baseos 253 k 2023-06-23T01:10:29.165 INFO:teuthology.orchestra.run.smithi123.stdout: libmount x86_64 2.32.1-42.el8 baseos 236 k 2023-06-23T01:10:29.165 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:10:29.165 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T01:10:29.165 INFO:teuthology.orchestra.run.smithi123.stdout: libsmartcols x86_64 2.32.1-42.el8 baseos 179 k 2023-06-23T01:10:29.165 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid x86_64 2.32.1-42.el8 baseos 98 k 2023-06-23T01:10:29.165 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux x86_64 2.32.1-42.el8 baseos 2.6 M 2023-06-23T01:10:29.165 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-user x86_64 2.32.1-42.el8 baseos 102 k 2023-06-23T01:10:29.166 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2023-06-23T01:10:29.166 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-06-23T01:10:29.166 INFO:teuthology.orchestra.run.smithi123.stdout: automake noarch 1.16.1-7.el8 appstream 713 k 2023-06-23T01:10:29.166 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T01:10:29.166 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T01:10:29.166 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T01:10:29.166 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T01:10:29.167 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T01:10:29.167 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:10:29.167 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T01:10:29.167 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T01:10:29.167 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-06-23T01:10:29.167 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T01:10:29.167 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-06-23T01:10:29.167 INFO:teuthology.orchestra.run.smithi123.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2023-06-23T01:10:29.168 INFO:teuthology.orchestra.run.smithi123.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2023-06-23T01:10:29.168 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T01:10:29.168 INFO:teuthology.orchestra.run.smithi123.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2023-06-23T01:10:29.168 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 8.5.0-19.el8 baseos 645 k 2023-06-23T01:10:29.168 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T01:10:29.168 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 8.5.0-19.el8 baseos 172 k 2023-06-23T01:10:29.168 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.9.0-3.el8 baseos 47 k 2023-06-23T01:10:29.169 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T01:10:29.169 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-06-23T01:10:29.169 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T01:10:29.169 INFO:teuthology.orchestra.run.smithi123.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2023-06-23T01:10:29.169 INFO:teuthology.orchestra.run.smithi123.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2023-06-23T01:10:29.169 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T01:10:29.169 INFO:teuthology.orchestra.run.smithi123.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2023-06-23T01:10:29.170 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2023-06-23T01:10:29.170 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-06-23T01:10:29.170 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-06-23T01:10:29.170 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-06-23T01:10:29.170 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-06-23T01:10:29.170 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2023-06-23T01:10:29.170 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2023-06-23T01:10:29.170 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2023-06-23T01:10:29.171 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T01:10:29.171 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-06-23T01:10:29.171 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T01:10:29.171 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T01:10:29.171 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T01:10:29.171 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T01:10:29.171 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T01:10:29.171 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T01:10:29.172 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-06-23T01:10:29.172 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T01:10:29.172 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T01:10:29.172 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-06-23T01:10:29.172 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T01:10:29.172 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T01:10:29.172 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T01:10:29.173 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-06-23T01:10:29.173 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-06-23T01:10:29.173 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-06-23T01:10:29.173 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-06-23T01:10:29.173 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T01:10:29.173 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T01:10:29.173 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-06-23T01:10:29.173 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-06-23T01:10:29.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T01:10:29.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T01:10:29.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-06-23T01:10:29.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-06-23T01:10:29.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-06-23T01:10:29.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T01:10:29.174 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T01:10:29.175 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-06-23T01:10:29.175 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-06-23T01:10:29.175 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-06-23T01:10:29.175 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-06-23T01:10:29.175 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-06-23T01:10:29.175 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2017.2-9.el8 appstream 54 k 2023-06-23T01:10:29.175 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T01:10:29.175 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-06-23T01:10:29.176 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T01:10:29.176 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T01:10:29.176 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-06-23T01:10:29.176 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T01:10:29.176 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T01:10:29.176 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T01:10:29.176 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T01:10:29.176 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T01:10:29.177 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T01:10:29.177 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T01:10:29.177 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T01:10:29.177 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-06-23T01:10:29.177 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T01:10:29.177 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-06-23T01:10:29.177 INFO:teuthology.orchestra.run.smithi123.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T01:10:29.178 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-06-23T01:10:29.178 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2023-06-23T01:10:29.178 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:10:29.178 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-06-23T01:10:29.178 INFO:teuthology.orchestra.run.smithi123.stdout:==================================================================================================== 2023-06-23T01:10:29.178 INFO:teuthology.orchestra.run.smithi123.stdout:Install 123 Packages 2023-06-23T01:10:29.178 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 15 Packages 2023-06-23T01:10:29.178 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:10:29.179 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 209 M 2023-06-23T01:10:29.179 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-06-23T01:10:30.083 INFO:teuthology.orchestra.run.smithi123.stdout:(1/138): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 35 kB/s | 6.4 kB 00:00 2023-06-23T01:10:30.442 INFO:teuthology.orchestra.run.smithi123.stdout:(2/138): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 2.4 MB/s | 873 kB 00:00 2023-06-23T01:10:30.525 INFO:teuthology.orchestra.run.smithi123.stdout:(3/138): ceph-immutable-object-cache-17.2.6-719 1.9 MB/s | 162 kB 00:00 2023-06-23T01:10:30.684 INFO:teuthology.orchestra.run.smithi123.stdout:(4/138): ceph-base-17.2.6-719.g42ddc46e.el8.x86 6.6 MB/s | 5.2 MB 00:00 2023-06-23T01:10:30.876 INFO:teuthology.orchestra.run.smithi123.stdout:(5/138): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 8.2 MB/s | 1.6 MB 00:00 2023-06-23T01:10:31.084 INFO:teuthology.orchestra.run.smithi123.stdout:(6/138): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 4.1 MB/s | 2.3 MB 00:00 2023-06-23T01:10:31.285 INFO:teuthology.orchestra.run.smithi123.stdout:(7/138): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 9.6 MB/s | 3.9 MB 00:00 2023-06-23T01:10:32.276 INFO:teuthology.orchestra.run.smithi123.stdout:(8/138): ceph-common-17.2.6-719.g42ddc46e.el8.x 9.8 MB/s | 23 MB 00:02 2023-06-23T01:10:32.360 INFO:teuthology.orchestra.run.smithi123.stdout:(9/138): ceph-selinux-17.2.6-719.g42ddc46e.el8. 291 kB/s | 24 kB 00:00 2023-06-23T01:10:32.794 INFO:teuthology.orchestra.run.smithi123.stdout:(10/138): ceph-radosgw-17.2.6-719.g42ddc46e.el8 11 MB/s | 17 MB 00:01 2023-06-23T01:10:32.877 INFO:teuthology.orchestra.run.smithi123.stdout:(11/138): cephfs-mirror-17.2.6-719.g42ddc46e.el 3.0 MB/s | 255 kB 00:00 2023-06-23T01:10:32.953 INFO:teuthology.orchestra.run.smithi123.stdout:(12/138): libcephfs-devel-17.2.6-719.g42ddc46e. 331 kB/s | 25 kB 00:00 2023-06-23T01:10:33.061 INFO:teuthology.orchestra.run.smithi123.stdout:(13/138): libcephfs2-17.2.6-719.g42ddc46e.el8.x 7.2 MB/s | 794 kB 00:00 2023-06-23T01:10:33.145 INFO:teuthology.orchestra.run.smithi123.stdout:(14/138): libcephsqlite-17.2.6-719.g42ddc46e.el 2.0 MB/s | 174 kB 00:00 2023-06-23T01:10:33.228 INFO:teuthology.orchestra.run.smithi123.stdout:(15/138): librados-devel-17.2.6-719.g42ddc46e.e 1.5 MB/s | 130 kB 00:00 2023-06-23T01:10:33.329 INFO:teuthology.orchestra.run.smithi123.stdout:(16/138): libradosstriper1-17.2.6-719.g42ddc46e 5.0 MB/s | 506 kB 00:00 2023-06-23T01:10:33.937 INFO:teuthology.orchestra.run.smithi123.stdout:(17/138): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 9.1 MB/s | 5.5 MB 00:00 2023-06-23T01:10:34.038 INFO:teuthology.orchestra.run.smithi123.stdout:(18/138): python3-ceph-argparse-17.2.6-719.g42d 445 kB/s | 45 kB 00:00 2023-06-23T01:10:34.213 INFO:teuthology.orchestra.run.smithi123.stdout:(19/138): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 5.3 MB/s | 17 MB 00:03 2023-06-23T01:10:34.238 INFO:teuthology.orchestra.run.smithi123.stdout:(20/138): python3-ceph-common-17.2.6-719.g42ddc 576 kB/s | 115 kB 00:00 2023-06-23T01:10:34.280 INFO:teuthology.orchestra.run.smithi123.stdout:(21/138): python3-cephfs-17.2.6-719.g42ddc46e.e 2.8 MB/s | 190 kB 00:00 2023-06-23T01:10:34.322 INFO:teuthology.orchestra.run.smithi123.stdout:(22/138): python3-rados-17.2.6-719.g42ddc46e.el 4.3 MB/s | 371 kB 00:00 2023-06-23T01:10:34.372 INFO:teuthology.orchestra.run.smithi123.stdout:(23/138): python3-rbd-17.2.6-719.g42ddc46e.el8. 3.8 MB/s | 357 kB 00:00 2023-06-23T01:10:34.406 INFO:teuthology.orchestra.run.smithi123.stdout:(24/138): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.2 MB/s | 107 kB 00:00 2023-06-23T01:10:34.448 INFO:teuthology.orchestra.run.smithi123.stdout:(25/138): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.2 MB/s | 91 kB 00:00 2023-06-23T01:10:34.557 INFO:teuthology.orchestra.run.smithi123.stdout:(26/138): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 1.6 MB/s | 173 kB 00:00 2023-06-23T01:10:34.666 INFO:teuthology.orchestra.run.smithi123.stdout:(27/138): ceph-grafana-dashboards-17.2.6-719.g4 213 kB/s | 23 kB 00:00 2023-06-23T01:10:34.757 INFO:teuthology.orchestra.run.smithi123.stdout:(28/138): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 9.9 MB/s | 3.5 MB 00:00 2023-06-23T01:10:34.782 INFO:teuthology.orchestra.run.smithi123.stdout:(29/138): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 1.0 MB/s | 121 kB 00:00 2023-06-23T01:10:34.958 INFO:teuthology.orchestra.run.smithi123.stdout:(30/138): ceph-mgr-dashboard-17.2.6-719.g42ddc4 8.5 MB/s | 1.7 MB 00:00 2023-06-23T01:10:35.050 INFO:teuthology.orchestra.run.smithi123.stdout:(31/138): ceph-mgr-modules-core-17.2.6-719.g42d 2.5 MB/s | 238 kB 00:00 2023-06-23T01:10:35.133 INFO:teuthology.orchestra.run.smithi123.stdout:(32/138): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 568 kB/s | 47 kB 00:00 2023-06-23T01:10:35.225 INFO:teuthology.orchestra.run.smithi123.stdout:(33/138): ceph-prometheus-alerts-17.2.6-719.g42 158 kB/s | 14 kB 00:00 2023-06-23T01:10:35.326 INFO:teuthology.orchestra.run.smithi123.stdout:(34/138): ceph-volume-17.2.6-719.g42ddc46e.el8. 2.6 MB/s | 264 kB 00:00 2023-06-23T01:10:35.417 INFO:teuthology.orchestra.run.smithi123.stdout:(35/138): cephadm-17.2.6-719.g42ddc46e.el8.noar 875 kB/s | 80 kB 00:00 2023-06-23T01:10:35.860 INFO:teuthology.orchestra.run.smithi123.stdout:(36/138): ceph-test-17.2.6-719.g42ddc46e.el8.x8 15 MB/s | 52 MB 00:03 2023-06-23T01:10:35.876 INFO:teuthology.orchestra.run.smithi123.stdout:(37/138): cephfs-top-17.2.6-719.g42ddc46e.el8.n 39 kB/s | 18 kB 00:00 2023-06-23T01:10:36.052 INFO:teuthology.orchestra.run.smithi123.stdout:(38/138): ceph-mgr-diskprediction-local-17.2.6- 5.8 MB/s | 7.4 MB 00:01 2023-06-23T01:10:36.194 INFO:teuthology.orchestra.run.smithi123.stdout:(39/138): bison-3.0.4-10.el8.x86_64.rpm 2.1 MB/s | 688 kB 00:00 2023-06-23T01:10:36.228 INFO:teuthology.orchestra.run.smithi123.stdout:(40/138): automake-1.16.1-7.el8.noarch.rpm 1.9 MB/s | 713 kB 00:00 2023-06-23T01:10:36.253 INFO:teuthology.orchestra.run.smithi123.stdout:(41/138): flex-2.6.1-9.el8.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2023-06-23T01:10:36.270 INFO:teuthology.orchestra.run.smithi123.stdout:(42/138): jq-1.6-7.el8.x86_64.rpm 2.6 MB/s | 202 kB 00:00 2023-06-23T01:10:36.312 INFO:teuthology.orchestra.run.smithi123.stdout:(43/138): librdkafka-0.11.4-3.el8.x86_64.rpm 4.1 MB/s | 354 kB 00:00 2023-06-23T01:10:36.354 INFO:teuthology.orchestra.run.smithi123.stdout:(44/138): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.0 MB/s | 259 kB 00:00 2023-06-23T01:10:36.379 INFO:teuthology.orchestra.run.smithi123.stdout:(45/138): oniguruma-6.8.2-2.el8.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2023-06-23T01:10:36.404 INFO:teuthology.orchestra.run.smithi123.stdout:(46/138): libtool-2.4.6-25.el8.x86_64.rpm 4.6 MB/s | 709 kB 00:00 2023-06-23T01:10:37.063 INFO:teuthology.orchestra.run.smithi123.stdout:(47/138): python3-babel-2.5.1-7.el8.noarch.rpm 7.3 MB/s | 4.8 MB 00:00 2023-06-23T01:10:37.122 INFO:teuthology.orchestra.run.smithi123.stdout:(48/138): openblas-threads-0.3.15-4.el8.x86_64. 6.5 MB/s | 4.9 MB 00:00 2023-06-23T01:10:37.173 INFO:teuthology.orchestra.run.smithi123.stdout:(49/138): python3-jsonpatch-1.21-2.el8.noarch.r 534 kB/s | 27 kB 00:00 2023-06-23T01:10:37.222 INFO:teuthology.orchestra.run.smithi123.stdout:(50/138): python3-jinja2-2.10.1-3.el8.noarch.rp 3.3 MB/s | 538 kB 00:00 2023-06-23T01:10:37.247 INFO:teuthology.orchestra.run.smithi123.stdout:(51/138): python3-jsonpointer-1.10-11.el8.noarc 266 kB/s | 20 kB 00:00 2023-06-23T01:10:37.298 INFO:teuthology.orchestra.run.smithi123.stdout:(52/138): python3-mako-1.0.6-14.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2023-06-23T01:10:37.348 INFO:teuthology.orchestra.run.smithi123.stdout:(53/138): python3-markupsafe-0.23-19.el8.x86_64 387 kB/s | 39 kB 00:00 2023-06-23T01:10:37.407 INFO:teuthology.orchestra.run.smithi123.stdout:(54/138): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.8 MB/s | 225 kB 00:00 2023-06-23T01:10:37.465 INFO:teuthology.orchestra.run.smithi123.stdout:(55/138): python3-prettytable-0.7.2-14.el8.noar 754 kB/s | 44 kB 00:00 2023-06-23T01:10:37.524 INFO:teuthology.orchestra.run.smithi123.stdout:(56/138): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.7 MB/s | 103 kB 00:00 2023-06-23T01:10:37.582 INFO:teuthology.orchestra.run.smithi123.stdout:(57/138): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.1 MB/s | 126 kB 00:00 2023-06-23T01:10:37.641 INFO:teuthology.orchestra.run.smithi123.stdout:(58/138): python3-pyasn1-modules-0.3.7-6.el8.no 1.8 MB/s | 110 kB 00:00 2023-06-23T01:10:37.699 INFO:teuthology.orchestra.run.smithi123.stdout:(59/138): python3-pytz-2017.2-9.el8.noarch.rpm 932 kB/s | 54 kB 00:00 2023-06-23T01:10:37.817 INFO:teuthology.orchestra.run.smithi123.stdout:(60/138): python3-numpy-1.14.3-10.el8.x86_64.rp 7.2 MB/s | 3.7 MB 00:00 2023-06-23T01:10:37.908 INFO:teuthology.orchestra.run.smithi123.stdout:(61/138): python3-werkzeug-0.12.2-4.el8.noarch. 4.9 MB/s | 457 kB 00:00 2023-06-23T01:10:37.992 INFO:teuthology.orchestra.run.smithi123.stdout:(62/138): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-06-23T01:10:38.043 INFO:teuthology.orchestra.run.smithi123.stdout:(63/138): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2023-06-23T01:10:38.176 INFO:teuthology.orchestra.run.smithi123.stdout:(64/138): elfutils-libelf-devel-0.189-2.el8.x86 462 kB/s | 61 kB 00:00 2023-06-23T01:10:38.259 INFO:teuthology.orchestra.run.smithi123.stdout:(65/138): fuse-2.9.7-17.el8.x86_64.rpm 996 kB/s | 83 kB 00:00 2023-06-23T01:10:38.318 INFO:teuthology.orchestra.run.smithi123.stdout:(66/138): keyutils-libs-devel-1.5.10-9.el8.x86_ 826 kB/s | 48 kB 00:00 2023-06-23T01:10:38.460 INFO:teuthology.orchestra.run.smithi123.stdout:(67/138): krb5-devel-1.18.2-22.el8.x86_64.rpm 3.9 MB/s | 561 kB 00:00 2023-06-23T01:10:38.545 INFO:teuthology.orchestra.run.smithi123.stdout:(68/138): libaio-devel-0.3.112-1.el8.x86_64.rpm 220 kB/s | 19 kB 00:00 2023-06-23T01:10:38.636 INFO:teuthology.orchestra.run.smithi123.stdout:(69/138): openblas-0.3.15-4.el8.x86_64.rpm 2.1 MB/s | 4.7 MB 00:02 2023-06-23T01:10:38.661 INFO:teuthology.orchestra.run.smithi123.stdout:(70/138): libcom_err-devel-1.45.6-5.el8.x86_64. 334 kB/s | 39 kB 00:00 2023-06-23T01:10:38.770 INFO:teuthology.orchestra.run.smithi123.stdout:(71/138): libquadmath-8.5.0-19.el8.x86_64.rpm 1.6 MB/s | 172 kB 00:00 2023-06-23T01:10:38.820 INFO:teuthology.orchestra.run.smithi123.stdout:(72/138): librabbitmq-0.9.0-3.el8.x86_64.rpm 939 kB/s | 47 kB 00:00 2023-06-23T01:10:38.845 INFO:teuthology.orchestra.run.smithi123.stdout:(73/138): libgfortran-8.5.0-19.el8.x86_64.rpm 3.0 MB/s | 645 kB 00:00 2023-06-23T01:10:38.904 INFO:teuthology.orchestra.run.smithi123.stdout:(74/138): libsepol-devel-2.9-3.el8.x86_64.rpm 1.5 MB/s | 87 kB 00:00 2023-06-23T01:10:38.938 INFO:teuthology.orchestra.run.smithi123.stdout:(75/138): libselinux-devel-2.9-8.el8.x86_64.rpm 1.7 MB/s | 200 kB 00:00 2023-06-23T01:10:38.980 INFO:teuthology.orchestra.run.smithi123.stdout:(76/138): libverto-devel-0.3.2-2.el8.x86_64.rpm 433 kB/s | 18 kB 00:00 2023-06-23T01:10:39.030 INFO:teuthology.orchestra.run.smithi123.stdout:(77/138): libzstd-devel-1.4.4-1.el8.x86_64.rpm 877 kB/s | 44 kB 00:00 2023-06-23T01:10:39.347 INFO:teuthology.orchestra.run.smithi123.stdout:(78/138): libuuid-devel-2.32.1-42.el8.x86_64.rp 224 kB/s | 99 kB 00:00 2023-06-23T01:10:39.555 INFO:teuthology.orchestra.run.smithi123.stdout:(79/138): pcre2-devel-10.32-3.el8.x86_64.rpm 2.8 MB/s | 605 kB 00:00 2023-06-23T01:10:39.947 INFO:teuthology.orchestra.run.smithi123.stdout:(80/138): pcre2-utf16-10.32-3.el8.x86_64.rpm 585 kB/s | 229 kB 00:00 2023-06-23T01:10:40.356 INFO:teuthology.orchestra.run.smithi123.stdout:(81/138): pcre2-utf32-10.32-3.el8.x86_64.rpm 540 kB/s | 220 kB 00:00 2023-06-23T01:10:40.465 INFO:teuthology.orchestra.run.smithi123.stdout:(82/138): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.2 MB/s | 249 kB 00:00 2023-06-23T01:10:41.140 INFO:teuthology.orchestra.run.smithi123.stdout:(83/138): python3-scipy-1.0.0-21.module_el8.5.0 4.0 MB/s | 14 MB 00:03 2023-06-23T01:10:41.273 INFO:teuthology.orchestra.run.smithi123.stdout:(84/138): python3-jwt-1.6.1-2.el8.noarch.rpm 321 kB/s | 43 kB 00:00 2023-06-23T01:10:41.540 INFO:teuthology.orchestra.run.smithi123.stdout:(85/138): openssl-devel-1.1.1k-9.el8.x86_64.rpm 1.3 MB/s | 3.2 MB 00:02 2023-06-23T01:10:41.632 INFO:teuthology.orchestra.run.smithi123.stdout:(86/138): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2023-06-23T01:10:41.657 INFO:teuthology.orchestra.run.smithi123.stdout:(87/138): python3-oauthlib-2.1.0-1.el8.noarch.r 404 kB/s | 155 kB 00:00 2023-06-23T01:10:41.691 INFO:teuthology.orchestra.run.smithi123.stdout:(88/138): python3-requests-oauthlib-1.0.0-1.el8 729 kB/s | 43 kB 00:00 2023-06-23T01:10:41.716 INFO:teuthology.orchestra.run.smithi123.stdout:(89/138): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.2 MB/s | 74 kB 00:00 2023-06-23T01:10:41.733 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:10:41.736 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:10:41.766 INFO:teuthology.orchestra.run.smithi123.stdout:(90/138): zlib-devel-1.2.11-21.el8.x86_64.rpm 772 kB/s | 58 kB 00:00 2023-06-23T01:10:41.767 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:10:41.768 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:10:42.058 INFO:teuthology.orchestra.run.smithi123.stdout:(91/138): python3-cryptography-3.2.1-6.el8.x86_ 402 kB/s | 641 kB 00:01 2023-06-23T01:10:42.059 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:10:42.061 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:10:42.367 INFO:teuthology.orchestra.run.smithi123.stdout:(92/138): liboath-2.6.2-3.el8.x86_64.rpm 98 kB/s | 59 kB 00:00 2023-06-23T01:10:42.368 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T01:10:42.369 INFO:teuthology.orchestra.run.smithi123.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T01:10:42.634 INFO:teuthology.orchestra.run.smithi123.stdout:(93/138): gperftools-libs-2.7-9.el8.x86_64.rpm 334 kB/s | 306 kB 00:00 2023-06-23T01:10:42.659 INFO:teuthology.orchestra.run.smithi123.stdout:(94/138): libunwind-1.3.1-3.el8.x86_64.rpm 125 kB/s | 75 kB 00:00 2023-06-23T01:10:42.743 INFO:teuthology.orchestra.run.smithi123.stdout:(95/138): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 401 kB/s | 44 kB 00:00 2023-06-23T01:10:42.852 INFO:teuthology.orchestra.run.smithi123.stdout:(96/138): python3-cachetools-3.1.1-4.el8.noarch 304 kB/s | 33 kB 00:00 2023-06-23T01:10:42.952 INFO:teuthology.orchestra.run.smithi123.stdout:(97/138): python3-beautifulsoup4-4.6.3-2.el8.1. 633 kB/s | 185 kB 00:00 2023-06-23T01:10:42.977 INFO:teuthology.orchestra.run.smithi123.stdout:(98/138): python3-certifi-2018.10.15-7.el8.noar 130 kB/s | 16 kB 00:00 2023-06-23T01:10:43.019 INFO:teuthology.orchestra.run.smithi123.stdout:(99/138): python3-asyncssh-2.7.0-2.el8.noarch.r 730 kB/s | 476 kB 00:00 2023-06-23T01:10:43.069 INFO:teuthology.orchestra.run.smithi123.stdout:(100/138): python3-cheroot-8.5.2-1.el8.noarch.r 1.4 MB/s | 173 kB 00:00 2023-06-23T01:10:43.136 INFO:teuthology.orchestra.run.smithi123.stdout:(101/138): python3-google-auth-1.1.1-10.el8.noa 695 kB/s | 81 kB 00:00 2023-06-23T01:10:43.186 INFO:teuthology.orchestra.run.smithi123.stdout:(102/138): python3-jaraco-6.2-6.el8.noarch.rpm 95 kB/s | 11 kB 00:00 2023-06-23T01:10:43.254 INFO:teuthology.orchestra.run.smithi123.stdout:(103/138): python3-jaraco-functools-2.0-4.el8.n 150 kB/s | 18 kB 00:00 2023-06-23T01:10:43.370 INFO:teuthology.orchestra.run.smithi123.stdout:(104/138): python3-logutils-0.3.5-11.el8.noarch 418 kB/s | 49 kB 00:00 2023-06-23T01:10:43.487 INFO:teuthology.orchestra.run.smithi123.stdout:(105/138): python3-more-itertools-7.2.0-3.el8.n 503 kB/s | 59 kB 00:00 2023-06-23T01:10:43.521 INFO:teuthology.orchestra.run.smithi123.stdout:(106/138): python3-kubernetes-11.0.0-6.el8.noar 3.5 MB/s | 1.2 MB 00:00 2023-06-23T01:10:43.546 INFO:teuthology.orchestra.run.smithi123.stdout:(107/138): python3-cherrypy-18.4.0-1.el8.noarch 676 kB/s | 384 kB 00:00 2023-06-23T01:10:43.605 INFO:teuthology.orchestra.run.smithi123.stdout:(108/138): python3-natsort-7.1.1-2.el8.noarch.r 514 kB/s | 60 kB 00:00 2023-06-23T01:10:43.639 INFO:teuthology.orchestra.run.smithi123.stdout:(109/138): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2023-06-23T01:10:43.664 INFO:teuthology.orchestra.run.smithi123.stdout:(110/138): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2023-06-23T01:10:43.714 INFO:teuthology.orchestra.run.smithi123.stdout:(111/138): python3-repoze-lru-0.7-6.el8.noarch. 309 kB/s | 34 kB 00:00 2023-06-23T01:10:43.756 INFO:teuthology.orchestra.run.smithi123.stdout:(112/138): python3-routes-2.4.1-12.el8.noarch.r 1.6 MB/s | 196 kB 00:00 2023-06-23T01:10:43.781 INFO:teuthology.orchestra.run.smithi123.stdout:(113/138): python3-rsa-4.8-1.el8.noarch.rpm 525 kB/s | 61 kB 00:00 2023-06-23T01:10:43.815 INFO:teuthology.orchestra.run.smithi123.stdout:(114/138): python3-simplegeneric-0.8.1-17.el8.n 186 kB/s | 19 kB 00:00 2023-06-23T01:10:43.865 INFO:teuthology.orchestra.run.smithi123.stdout:(115/138): python3-singledispatch-3.4.0.3-18.el 219 kB/s | 24 kB 00:00 2023-06-23T01:10:43.882 INFO:teuthology.orchestra.run.smithi123.stdout:(116/138): python3-tempora-1.14.1-5.el8.noarch. 290 kB/s | 29 kB 00:00 2023-06-23T01:10:43.915 INFO:teuthology.orchestra.run.smithi123.stdout:(117/138): python3-trustme-0.6.0-4.el8.noarch.r 274 kB/s | 27 kB 00:00 2023-06-23T01:10:43.983 INFO:teuthology.orchestra.run.smithi123.stdout:(118/138): python3-waitress-1.4.3-1.el8.noarch. 2.1 MB/s | 255 kB 00:00 2023-06-23T01:10:44.024 INFO:teuthology.orchestra.run.smithi123.stdout:(119/138): python3-websocket-client-0.56.0-5.el 563 kB/s | 61 kB 00:00 2023-06-23T01:10:44.191 INFO:teuthology.orchestra.run.smithi123.stdout:(120/138): python3-webob-1.8.5-1.el8.1.noarch.r 812 kB/s | 251 kB 00:00 2023-06-23T01:10:44.208 INFO:teuthology.orchestra.run.smithi123.stdout:(121/138): python3-zc-lockfile-2.0-2.el8.noarch 125 kB/s | 23 kB 00:00 2023-06-23T01:10:44.234 INFO:teuthology.orchestra.run.smithi123.stdout:(122/138): python3-webtest-2.0.33-1.el8.noarch. 340 kB/s | 85 kB 00:00 2023-06-23T01:10:44.501 INFO:teuthology.orchestra.run.smithi123.stdout:(123/138): thrift-0.13.0-2.el8.x86_64.rpm 5.6 MB/s | 1.7 MB 00:00 2023-06-23T01:10:44.668 INFO:teuthology.orchestra.run.smithi123.stdout:(124/138): librbd1-17.2.6-719.g42ddc46e.el8.x86 8.3 MB/s | 3.6 MB 00:00 2023-06-23T01:10:44.693 INFO:teuthology.orchestra.run.smithi123.stdout:(125/138): NetworkManager-initscripts-updown-1. 746 kB/s | 143 kB 00:00 2023-06-23T01:10:44.752 INFO:teuthology.orchestra.run.smithi123.stdout:(126/138): librados2-17.2.6-719.g42ddc46e.el8.x 6.4 MB/s | 3.5 MB 00:00 2023-06-23T01:10:44.794 INFO:teuthology.orchestra.run.smithi123.stdout:(127/138): elfutils-debuginfod-client-0.189-2.e 603 kB/s | 76 kB 00:00 2023-06-23T01:10:44.844 INFO:teuthology.orchestra.run.smithi123.stdout:(128/138): elfutils-libelf-0.189-2.el8.x86_64.r 1.5 MB/s | 232 kB 00:00 2023-06-23T01:10:44.953 INFO:teuthology.orchestra.run.smithi123.stdout:(129/138): elfutils-libs-0.189-2.el8.x86_64.rpm 1.5 MB/s | 302 kB 00:00 2023-06-23T01:10:44.978 INFO:teuthology.orchestra.run.smithi123.stdout:(130/138): libblkid-2.32.1-42.el8.x86_64.rpm 1.6 MB/s | 220 kB 00:00 2023-06-23T01:10:45.086 INFO:teuthology.orchestra.run.smithi123.stdout:(131/138): libfdisk-2.32.1-42.el8.x86_64.rpm 1.9 MB/s | 253 kB 00:00 2023-06-23T01:10:45.111 INFO:teuthology.orchestra.run.smithi123.stdout:(132/138): libmount-2.32.1-42.el8.x86_64.rpm 1.7 MB/s | 236 kB 00:00 2023-06-23T01:10:45.195 INFO:teuthology.orchestra.run.smithi123.stdout:(133/138): libsmartcols-2.32.1-42.el8.x86_64.rp 1.6 MB/s | 179 kB 00:00 2023-06-23T01:10:45.221 INFO:teuthology.orchestra.run.smithi123.stdout:(134/138): libuuid-2.32.1-42.el8.x86_64.rpm 907 kB/s | 98 kB 00:00 2023-06-23T01:10:45.304 INFO:teuthology.orchestra.run.smithi123.stdout:(135/138): util-linux-user-2.32.1-42.el8.x86_64 1.2 MB/s | 102 kB 00:00 2023-06-23T01:10:45.438 INFO:teuthology.orchestra.run.smithi123.stdout:(136/138): iproute-6.2.0-2.el8.x86_64.rpm 1.3 MB/s | 880 kB 00:00 2023-06-23T01:10:45.713 INFO:teuthology.orchestra.run.smithi123.stdout:(137/138): xfsprogs-5.0.0-12.el8.x86_64.rpm 2.6 MB/s | 1.1 MB 00:00 2023-06-23T01:10:46.021 INFO:teuthology.orchestra.run.smithi123.stdout:(138/138): util-linux-2.32.1-42.el8.x86_64.rpm 3.1 MB/s | 2.6 MB 00:00 2023-06-23T01:10:46.028 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:10:46.029 INFO:teuthology.orchestra.run.smithi123.stdout:Total 12 MB/s | 209 MB 00:16 2023-06-23T01:10:46.836 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-06-23T01:10:47.259 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-06-23T01:10:47.259 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-06-23T01:10:48.700 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-06-23T01:10:48.703 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-06-23T01:10:53.947 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-06-23T01:10:53.984 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libuuid-2.32.1-42.el8.x86_64 1/1 2023-06-23T01:10:54.124 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libuuid-2.32.1-42.el8.x86_64 1/153 2023-06-23T01:10:54.213 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libuuid-2.32.1-42.el8.x86_64 1/153 2023-06-23T01:10:54.382 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libblkid-2.32.1-42.el8.x86_64 2/153 2023-06-23T01:10:54.444 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libblkid-2.32.1-42.el8.x86_64 2/153 2023-06-23T01:10:54.668 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/153 2023-06-23T01:10:55.068 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-libelf-0.189-2.el8.x86_64 4/153 2023-06-23T01:10:55.279 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : thrift-0.13.0-2.el8.x86_64 5/153 2023-06-23T01:10:55.759 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 6/153 2023-06-23T01:10:55.924 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 7/153 2023-06-23T01:10:56.313 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 7/153 2023-06-23T01:10:56.540 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 8/153 2023-06-23T01:10:56.923 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 8/153 2023-06-23T01:10:57.156 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 9/153 2023-06-23T01:10:57.275 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 9/153 2023-06-23T01:10:57.490 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/153 2023-06-23T01:10:57.596 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/153 2023-06-23T01:10:57.830 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 11/153 2023-06-23T01:10:58.031 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 12/153 2023-06-23T01:10:58.588 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 12/153 2023-06-23T01:10:58.739 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 13/153 2023-06-23T01:10:58.817 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 13/153 2023-06-23T01:10:59.062 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 14/153 2023-06-23T01:10:59.313 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 15/153 2023-06-23T01:10:59.498 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/153 2023-06-23T01:10:59.604 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/153 2023-06-23T01:10:59.764 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 17/153 2023-06-23T01:10:59.964 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 17/153 2023-06-23T01:11:00.181 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 18/153 2023-06-23T01:11:00.293 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 18/153 2023-06-23T01:11:00.529 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 19/153 2023-06-23T01:11:00.738 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 20/153 2023-06-23T01:11:00.889 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-8.5.0-19.el8.x86_64 21/153 2023-06-23T01:11:01.026 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libquadmath-8.5.0-19.el8.x86_64 21/153 2023-06-23T01:11:01.223 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-8.5.0-19.el8.x86_64 22/153 2023-06-23T01:11:01.819 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libgfortran-8.5.0-19.el8.x86_64 22/153 2023-06-23T01:11:02.031 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.15-4.el8.x86_64 23/153 2023-06-23T01:11:02.649 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 23/153 2023-06-23T01:11:02.839 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-06-23T01:11:03.556 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-06-23T01:11:03.883 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/153 2023-06-23T01:11:04.120 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/153 2023-06-23T01:11:04.293 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 27/153 2023-06-23T01:11:04.498 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/153 2023-06-23T01:11:04.712 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 29/153 2023-06-23T01:11:04.891 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 30/153 2023-06-23T01:11:05.110 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 31/153 2023-06-23T01:11:06.106 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 32/153 2023-06-23T01:11:06.411 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/153 2023-06-23T01:11:06.702 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/153 2023-06-23T01:11:06.880 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-06-23T01:11:08.673 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-06-23T01:11:08.950 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 36/153 2023-06-23T01:11:09.114 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/153 2023-06-23T01:11:09.316 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.8-1.el8.noarch 38/153 2023-06-23T01:11:09.523 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 39/153 2023-06-23T01:11:09.694 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 40/153 2023-06-23T01:11:09.938 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 41/153 2023-06-23T01:11:10.111 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-libs-0.189-2.el8.x86_64 42/153 2023-06-23T01:11:10.331 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : elfutils-debuginfod-client-0.189-2.el8.x86_64 43/153 2023-06-23T01:11:10.538 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libfdisk-2.32.1-42.el8.x86_64 44/153 2023-06-23T01:11:10.598 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libfdisk-2.32.1-42.el8.x86_64 44/153 2023-06-23T01:11:10.821 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libmount-2.32.1-42.el8.x86_64 45/153 2023-06-23T01:11:10.873 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libmount-2.32.1-42.el8.x86_64 45/153 2023-06-23T01:11:11.032 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libuuid-devel-2.32.1-42.el8.x86_64 46/153 2023-06-23T01:11:11.238 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : libsmartcols-2.32.1-42.el8.x86_64 47/153 2023-06-23T01:11:11.742 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libsmartcols-2.32.1-42.el8.x86_64 47/153 2023-06-23T01:11:11.988 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : util-linux-2.32.1-42.el8.x86_64 48/153 2023-06-23T01:11:12.003 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: util-linux-2.32.1-42.el8.x86_64 48/153 2023-06-23T01:11:12.166 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 49/153 2023-06-23T01:11:12.431 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 50/153 2023-06-23T01:11:12.652 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 51/153 2023-06-23T01:11:12.808 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/153 2023-06-23T01:11:12.956 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/153 2023-06-23T01:11:13.146 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/153 2023-06-23T01:11:13.358 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/153 2023-06-23T01:11:13.573 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 56/153 2023-06-23T01:11:13.732 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 57/153 2023-06-23T01:11:13.960 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 58/153 2023-06-23T01:11:14.133 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 59/153 2023-06-23T01:11:14.302 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/153 2023-06-23T01:11:14.488 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/153 2023-06-23T01:11:14.671 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 62/153 2023-06-23T01:11:14.898 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 63/153 2023-06-23T01:11:15.145 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 64/153 2023-06-23T01:11:15.370 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 65/153 2023-06-23T01:11:15.726 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 66/153 2023-06-23T01:11:15.946 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 67/153 2023-06-23T01:11:16.130 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 68/153 2023-06-23T01:11:16.410 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 69/153 2023-06-23T01:11:16.658 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 70/153 2023-06-23T01:11:17.023 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 71/153 2023-06-23T01:11:17.265 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 72/153 2023-06-23T01:11:17.497 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 73/153 2023-06-23T01:11:17.749 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 74/153 2023-06-23T01:11:17.967 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 75/153 2023-06-23T01:11:19.002 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/153 2023-06-23T01:11:19.319 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 77/153 2023-06-23T01:11:19.573 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 78/153 2023-06-23T01:11:19.808 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 79/153 2023-06-23T01:11:20.120 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/153 2023-06-23T01:11:20.329 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/153 2023-06-23T01:11:20.572 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 82/153 2023-06-23T01:11:20.764 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 83/153 2023-06-23T01:11:21.051 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 84/153 2023-06-23T01:11:21.300 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 85/153 2023-06-23T01:11:21.490 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 86/153 2023-06-23T01:11:21.667 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 87/153 2023-06-23T01:11:21.977 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 88/153 2023-06-23T01:11:22.198 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 89/153 2023-06-23T01:11:22.390 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 90/153 2023-06-23T01:11:22.616 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/153 2023-06-23T01:11:22.837 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 92/153 2023-06-23T01:11:23.046 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.7-17.el8.x86_64 93/153 2023-06-23T01:11:23.265 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 94/153 2023-06-23T01:11:23.435 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 95/153 2023-06-23T01:11:23.620 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 96/153 2023-06-23T01:11:26.254 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 97/153 2023-06-23T01:11:26.519 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 97/153 2023-06-23T01:11:27.258 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 97/153 2023-06-23T01:11:27.544 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 98/153 2023-06-23T01:11:27.619 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 98/153 2023-06-23T01:11:27.793 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 99/153 2023-06-23T01:11:46.445 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 99/153 2023-06-23T01:11:46.446 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2023-06-23T01:11:46.446 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2023-06-23T01:11:46.446 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2023-06-23T01:11:46.446 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /run 2023-06-23T01:11:46.446 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2023-06-23T01:11:46.446 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2023-06-23T01:11:46.447 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2023-06-23T01:11:46.447 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2023-06-23T01:11:46.447 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2023-06-23T01:11:46.447 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:11:46.805 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 100/153 2023-06-23T01:11:48.268 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 100/153 2023-06-23T01:11:48.537 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 101/153 2023-06-23T01:11:48.750 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 101/153 2023-06-23T01:11:48.982 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 102/153 2023-06-23T01:11:49.368 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 102/153 2023-06-23T01:11:49.590 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 103/153 2023-06-23T01:11:49.614 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 103/153 2023-06-23T01:11:49.833 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 104/153 2023-06-23T01:11:50.025 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 105/153 2023-06-23T01:11:50.181 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 106/153 2023-06-23T01:11:50.246 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 106/153 2023-06-23T01:11:50.507 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-7.el8.x86_64 107/153 2023-06-23T01:11:50.749 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : automake-1.16.1-7.el8.noarch 108/153 2023-06-23T01:11:51.170 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 109/153 2023-06-23T01:11:51.342 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 109/153 2023-06-23T01:11:51.471 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 110/153 2023-06-23T01:11:51.659 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 111/153 2023-06-23T01:11:51.898 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 112/153 2023-06-23T01:11:52.107 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 112/153 2023-06-23T01:11:52.339 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 113/153 2023-06-23T01:11:53.300 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 113/153 2023-06-23T01:11:53.536 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 114/153 2023-06-23T01:11:53.628 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 114/153 2023-06-23T01:11:53.984 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 115/153 2023-06-23T01:11:54.264 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 116/153 2023-06-23T01:11:54.300 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 116/153 2023-06-23T01:11:54.506 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 117/153 2023-06-23T01:11:54.517 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 117/153 2023-06-23T01:11:54.719 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 118/153 2023-06-23T01:11:54.872 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/153 2023-06-23T01:11:59.758 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/153 2023-06-23T01:12:01.569 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 120/153 2023-06-23T01:12:01.771 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 121/153 2023-06-23T01:12:01.859 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 121/153 2023-06-23T01:12:02.103 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 122/153 2023-06-23T01:12:02.144 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 122/153 2023-06-23T01:12:02.355 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 123/153 2023-06-23T01:12:02.700 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 123/153 2023-06-23T01:12:02.913 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 124/153 2023-06-23T01:12:03.006 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 124/153 2023-06-23T01:12:03.865 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 125/153 2023-06-23T01:12:04.142 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openssl-devel-1:1.1.1k-9.el8.x86_64 126/153 2023-06-23T01:12:04.318 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : elfutils-libelf-devel-0.189-2.el8.x86_64 127/153 2023-06-23T01:12:04.507 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : util-linux-user-2.32.1-42.el8.x86_64 128/153 2023-06-23T01:12:04.674 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 129/153 2023-06-23T01:12:04.884 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 130/153 2023-06-23T01:12:05.112 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 131/153 2023-06-23T01:12:05.337 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 132/153 2023-06-23T01:12:05.672 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 133/153 2023-06-23T01:12:05.923 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : iproute-6.2.0-2.el8.x86_64 134/153 2023-06-23T01:12:06.052 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.16-7.el 135/153 2023-06-23T01:12:06.064 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-7.el 135/153 2023-06-23T01:12:06.243 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 136/153 2023-06-23T01:12:06.479 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : flex-2.6.1-9.el8.x86_64 137/153 2023-06-23T01:12:06.568 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 137/153 2023-06-23T01:12:06.785 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : bison-3.0.4-10.el8.x86_64 138/153 2023-06-23T01:12:06.797 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 138/153 2023-06-23T01:12:06.964 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 139/153 2023-06-23T01:12:07.136 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 140/153 2023-06-23T01:12:07.137 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 141/153 2023-06-23T01:12:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 141/153 2023-06-23T01:12:07.363 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 142/153 2023-06-23T01:12:07.620 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 142/153 2023-06-23T01:12:07.626 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-4.el 143/153 2023-06-23T01:12:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.40.16-4.el 143/153 2023-06-23T01:12:07.749 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 144/153 2023-06-23T01:12:08.001 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 144/153 2023-06-23T01:12:08.189 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 145/153 2023-06-23T01:12:08.359 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 146/153 2023-06-23T01:12:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 147/153 2023-06-23T01:12:08.572 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 148/153 2023-06-23T01:12:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 148/153 2023-06-23T01:12:08.798 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 149/153 2023-06-23T01:12:08.989 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 149/153 2023-06-23T01:12:08.990 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 150/153 2023-06-23T01:12:09.214 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 150/153 2023-06-23T01:12:09.215 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 151/153 2023-06-23T01:12:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 151/153 2023-06-23T01:12:09.414 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 152/153 2023-06-23T01:12:09.595 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 152/153 2023-06-23T01:12:09.793 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 153/153 2023-06-23T01:12:10.409 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: elfutils-libelf-0.189-1.el8.x86_64 153/153 2023-06-23T01:12:10.409 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/153 2023-06-23T01:12:10.409 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/153 2023-06-23T01:12:10.409 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/153 2023-06-23T01:12:10.409 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/153 2023-06-23T01:12:10.410 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/153 2023-06-23T01:12:10.410 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/153 2023-06-23T01:12:10.410 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/153 2023-06-23T01:12:10.410 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/153 2023-06-23T01:12:10.410 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/153 2023-06-23T01:12:10.410 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/153 2023-06-23T01:12:10.410 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/153 2023-06-23T01:12:10.411 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/153 2023-06-23T01:12:10.411 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/153 2023-06-23T01:12:10.411 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/153 2023-06-23T01:12:10.411 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/153 2023-06-23T01:12:10.411 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/153 2023-06-23T01:12:10.411 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/153 2023-06-23T01:12:10.412 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/153 2023-06-23T01:12:10.412 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/153 2023-06-23T01:12:10.412 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/153 2023-06-23T01:12:10.412 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/153 2023-06-23T01:12:10.413 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/153 2023-06-23T01:12:10.413 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/153 2023-06-23T01:12:10.413 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/153 2023-06-23T01:12:10.413 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/153 2023-06-23T01:12:10.414 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/153 2023-06-23T01:12:10.414 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/153 2023-06-23T01:12:10.414 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/153 2023-06-23T01:12:10.414 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/153 2023-06-23T01:12:10.414 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/153 2023-06-23T01:12:10.415 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/153 2023-06-23T01:12:10.415 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/153 2023-06-23T01:12:10.415 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/153 2023-06-23T01:12:10.415 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/153 2023-06-23T01:12:10.415 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/153 2023-06-23T01:12:10.415 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/153 2023-06-23T01:12:10.416 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/153 2023-06-23T01:12:10.416 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/153 2023-06-23T01:12:10.416 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : automake-1.16.1-7.el8.noarch 39/153 2023-06-23T01:12:10.416 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/153 2023-06-23T01:12:10.416 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/153 2023-06-23T01:12:10.416 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-7.el8.x86_64 42/153 2023-06-23T01:12:10.417 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 43/153 2023-06-23T01:12:10.417 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 44/153 2023-06-23T01:12:10.417 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 45/153 2023-06-23T01:12:10.417 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 46/153 2023-06-23T01:12:10.417 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 47/153 2023-06-23T01:12:10.417 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 48/153 2023-06-23T01:12:10.417 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 49/153 2023-06-23T01:12:10.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 50/153 2023-06-23T01:12:10.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 51/153 2023-06-23T01:12:10.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 52/153 2023-06-23T01:12:10.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 53/153 2023-06-23T01:12:10.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 54/153 2023-06-23T01:12:10.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 55/153 2023-06-23T01:12:10.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 56/153 2023-06-23T01:12:10.418 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 57/153 2023-06-23T01:12:10.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 58/153 2023-06-23T01:12:10.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 59/153 2023-06-23T01:12:10.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 60/153 2023-06-23T01:12:10.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 61/153 2023-06-23T01:12:10.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 62/153 2023-06-23T01:12:10.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 63/153 2023-06-23T01:12:10.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 64/153 2023-06-23T01:12:10.419 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 65/153 2023-06-23T01:12:10.420 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-devel-0.189-2.el8.x86_64 66/153 2023-06-23T01:12:10.420 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 67/153 2023-06-23T01:12:10.420 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 68/153 2023-06-23T01:12:10.420 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 69/153 2023-06-23T01:12:10.420 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 70/153 2023-06-23T01:12:10.421 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 71/153 2023-06-23T01:12:10.421 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-8.5.0-19.el8.x86_64 72/153 2023-06-23T01:12:10.421 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-8.5.0-19.el8.x86_64 73/153 2023-06-23T01:12:10.421 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 74/153 2023-06-23T01:12:10.421 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 75/153 2023-06-23T01:12:10.421 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 76/153 2023-06-23T01:12:10.421 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-devel-2.32.1-42.el8.x86_64 77/153 2023-06-23T01:12:10.422 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 78/153 2023-06-23T01:12:10.422 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/153 2023-06-23T01:12:10.422 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openssl-devel-1:1.1.1k-9.el8.x86_64 80/153 2023-06-23T01:12:10.422 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 81/153 2023-06-23T01:12:10.422 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 82/153 2023-06-23T01:12:10.423 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 83/153 2023-06-23T01:12:10.423 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 84/153 2023-06-23T01:12:10.423 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 85/153 2023-06-23T01:12:10.423 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 86/153 2023-06-23T01:12:10.423 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 87/153 2023-06-23T01:12:10.423 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 88/153 2023-06-23T01:12:10.423 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 89/153 2023-06-23T01:12:10.424 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 90/153 2023-06-23T01:12:10.424 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 91/153 2023-06-23T01:12:10.424 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 92/153 2023-06-23T01:12:10.424 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 93/153 2023-06-23T01:12:10.424 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 94/153 2023-06-23T01:12:10.425 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 95/153 2023-06-23T01:12:10.425 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 96/153 2023-06-23T01:12:10.425 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/153 2023-06-23T01:12:10.426 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 98/153 2023-06-23T01:12:10.426 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 99/153 2023-06-23T01:12:10.426 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 100/153 2023-06-23T01:12:10.426 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 101/153 2023-06-23T01:12:10.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 102/153 2023-06-23T01:12:10.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 103/153 2023-06-23T01:12:10.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 104/153 2023-06-23T01:12:10.427 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 105/153 2023-06-23T01:12:10.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 106/153 2023-06-23T01:12:10.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 107/153 2023-06-23T01:12:10.428 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 108/153 2023-06-23T01:12:10.429 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 109/153 2023-06-23T01:12:10.429 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-2.6-1.el8.noarch 110/153 2023-06-23T01:12:10.429 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 111/153 2023-06-23T01:12:10.429 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 112/153 2023-06-23T01:12:10.430 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 113/153 2023-06-23T01:12:10.430 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 114/153 2023-06-23T01:12:10.430 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 115/153 2023-06-23T01:12:10.430 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 116/153 2023-06-23T01:12:10.431 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 117/153 2023-06-23T01:12:10.431 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 118/153 2023-06-23T01:12:10.431 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 119/153 2023-06-23T01:12:10.432 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 120/153 2023-06-23T01:12:10.432 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 121/153 2023-06-23T01:12:10.432 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 122/153 2023-06-23T01:12:10.432 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 123/153 2023-06-23T01:12:10.432 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 124/153 2023-06-23T01:12:10.433 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 125/153 2023-06-23T01:12:10.433 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 126/153 2023-06-23T01:12:10.433 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 127/153 2023-06-23T01:12:10.434 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-7.el 128/153 2023-06-23T01:12:10.434 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-4.el 129/153 2023-06-23T01:12:10.434 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-debuginfod-client-0.189-2.el8.x86_64 130/153 2023-06-23T01:12:10.434 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 131/153 2023-06-23T01:12:10.435 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-0.189-2.el8.x86_64 132/153 2023-06-23T01:12:10.435 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 133/153 2023-06-23T01:12:10.435 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libs-0.189-2.el8.x86_64 134/153 2023-06-23T01:12:10.435 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 135/153 2023-06-23T01:12:10.435 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : iproute-6.2.0-2.el8.x86_64 136/153 2023-06-23T01:12:10.436 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 137/153 2023-06-23T01:12:10.436 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libblkid-2.32.1-42.el8.x86_64 138/153 2023-06-23T01:12:10.436 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 139/153 2023-06-23T01:12:10.436 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libfdisk-2.32.1-42.el8.x86_64 140/153 2023-06-23T01:12:10.437 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 141/153 2023-06-23T01:12:10.437 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libmount-2.32.1-42.el8.x86_64 142/153 2023-06-23T01:12:10.437 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 143/153 2023-06-23T01:12:10.437 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsmartcols-2.32.1-42.el8.x86_64 144/153 2023-06-23T01:12:10.438 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 145/153 2023-06-23T01:12:10.438 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-2.32.1-42.el8.x86_64 146/153 2023-06-23T01:12:10.438 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 147/153 2023-06-23T01:12:10.438 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-2.32.1-42.el8.x86_64 148/153 2023-06-23T01:12:10.438 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 149/153 2023-06-23T01:12:10.439 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-user-2.32.1-42.el8.x86_64 150/153 2023-06-23T01:12:10.439 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 151/153 2023-06-23T01:12:10.439 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 152/153 2023-06-23T01:12:12.393 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 153/153 2023-06-23T01:12:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:12:12.394 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2023-06-23T01:12:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: NetworkManager-initscripts-updown-1:1.40.16-7.el8.noarch 2023-06-23T01:12:12.394 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-debuginfod-client-0.189-2.el8.x86_64 2023-06-23T01:12:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-0.189-2.el8.x86_64 2023-06-23T01:12:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libs-0.189-2.el8.x86_64 2023-06-23T01:12:12.395 INFO:teuthology.orchestra.run.smithi123.stdout: iproute-6.2.0-2.el8.x86_64 2023-06-23T01:12:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: libblkid-2.32.1-42.el8.x86_64 2023-06-23T01:12:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: libfdisk-2.32.1-42.el8.x86_64 2023-06-23T01:12:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: libmount-2.32.1-42.el8.x86_64 2023-06-23T01:12:12.396 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.397 INFO:teuthology.orchestra.run.smithi123.stdout: libsmartcols-2.32.1-42.el8.x86_64 2023-06-23T01:12:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-2.32.1-42.el8.x86_64 2023-06-23T01:12:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-2.32.1-42.el8.x86_64 2023-06-23T01:12:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: util-linux-user-2.32.1-42.el8.x86_64 2023-06-23T01:12:12.398 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-5.0.0-12.el8.x86_64 2023-06-23T01:12:12.399 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-06-23T01:12:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T01:12:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T01:12:12.399 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.400 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.401 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.402 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.403 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:12:12.404 INFO:teuthology.orchestra.run.smithi123.stdout: elfutils-libelf-devel-0.189-2.el8.x86_64 2023-06-23T01:12:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T01:12:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: fuse-2.9.7-17.el8.x86_64 2023-06-23T01:12:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T01:12:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: jq-1.6-7.el8.x86_64 2023-06-23T01:12:12.405 INFO:teuthology.orchestra.run.smithi123.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-06-23T01:12:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-06-23T01:12:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T01:12:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.406 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-06-23T01:12:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-8.5.0-19.el8.x86_64 2023-06-23T01:12:12.407 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T01:12:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-8.5.0-19.el8.x86_64 2023-06-23T01:12:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T01:12:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-06-23T01:12:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.408 INFO:teuthology.orchestra.run.smithi123.stdout: libselinux-devel-2.9-8.el8.x86_64 2023-06-23T01:12:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-06-23T01:12:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T01:12:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T01:12:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: libuuid-devel-2.32.1-42.el8.x86_64 2023-06-23T01:12:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-06-23T01:12:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2023-06-23T01:12:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T01:12:12.409 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T01:12:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.15-4.el8.x86_64 2023-06-23T01:12:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-06-23T01:12:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: openssl-devel-1:1.1.1k-9.el8.x86_64 2023-06-23T01:12:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-06-23T01:12:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-06-23T01:12:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-06-23T01:12:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T01:12:12.410 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.5.1-7.el8.noarch 2023-06-23T01:12:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T01:12:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T01:12:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T01:12:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.411 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T01:12:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-06-23T01:12:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T01:12:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T01:12:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-06-23T01:12:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T01:12:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T01:12:12.412 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T01:12:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-06-23T01:12:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T01:12:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T01:12:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T01:12:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T01:12:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T01:12:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.0.6-14.el8.noarch 2023-06-23T01:12:12.413 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T01:12:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T01:12:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T01:12:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-06-23T01:12:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-06-23T01:12:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T01:12:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T01:12:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T01:12:12.414 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T01:12:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-06-23T01:12:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T01:12:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T01:12:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.14-14.el8.noarch 2023-06-23T01:12:12.415 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T01:12:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.416 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T01:12:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T01:12:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T01:12:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T01:12:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-06-23T01:12:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T01:12:12.417 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T01:12:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T01:12:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T01:12:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T01:12:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T01:12:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T01:12:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T01:12:12.418 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T01:12:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T01:12:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:12:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-1.el8.x86_64 2023-06-23T01:12:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T01:12:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2023-06-23T01:12:12.419 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T01:12:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: zlib-devel-1.2.11-21.el8.x86_64 2023-06-23T01:12:12.420 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:12:12.420 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-06-23T01:12:12.566 DEBUG:teuthology.parallel:result is None 2023-06-23T01:16:58.793 INFO:teuthology.orchestra.run.smithi057.stdout:Extra Packages for Enterprise Linux 41 kB/s | 16 MB 06:33 2023-06-23T01:17:02.776 INFO:teuthology.orchestra.run.smithi057.stdout:lab-extras 428 kB/s | 24 kB 00:00 2023-06-23T01:17:06.815 INFO:teuthology.orchestra.run.smithi057.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:17:06.816 INFO:teuthology.orchestra.run.smithi057.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:17:06.829 INFO:teuthology.orchestra.run.smithi057.stdout:Package NetworkManager-1:1.40.16-4.el8.x86_64 is already installed. 2023-06-23T01:17:06.829 INFO:teuthology.orchestra.run.smithi057.stdout:Package NetworkManager-initscripts-updown-1:1.40.16-4.el8.noarch is already installed. 2023-06-23T01:17:06.830 INFO:teuthology.orchestra.run.smithi057.stdout:Package iproute-6.2.0-1.el8.x86_64 is already installed. 2023-06-23T01:17:06.831 INFO:teuthology.orchestra.run.smithi057.stdout:Package util-linux-2.32.1-41.el8.x86_64 is already installed. 2023-06-23T01:17:06.832 INFO:teuthology.orchestra.run.smithi057.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T01:17:06.834 INFO:teuthology.orchestra.run.smithi057.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T01:17:06.837 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-06-23T01:17:06.838 INFO:teuthology.orchestra.run.smithi057.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-06-23T01:17:07.025 INFO:teuthology.orchestra.run.smithi057.stdout:Dependencies resolved. 2023-06-23T01:17:07.039 INFO:teuthology.orchestra.run.smithi057.stdout:==================================================================================================== 2023-06-23T01:17:07.039 INFO:teuthology.orchestra.run.smithi057.stdout: Package Arch Version Repository Size 2023-06-23T01:17:07.040 INFO:teuthology.orchestra.run.smithi057.stdout:==================================================================================================== 2023-06-23T01:17:07.040 INFO:teuthology.orchestra.run.smithi057.stdout:Installing: 2023-06-23T01:17:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: bison x86_64 3.0.4-10.el8 appstream 688 k 2023-06-23T01:17:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T01:17:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T01:17:07.040 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T01:17:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T01:17:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T01:17:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T01:17:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T01:17:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T01:17:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T01:17:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:17:07.041 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T01:17:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T01:17:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T01:17:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T01:17:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T01:17:07.042 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel x86_64 0.189-2.el8 baseos 61 k 2023-06-23T01:17:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: flex x86_64 2.6.1-9.el8 appstream 320 k 2023-06-23T01:17:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel x86_64 0.3.112-1.el8 baseos 19 k 2023-06-23T01:17:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T01:17:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T01:17:07.043 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T01:17:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: libtool x86_64 2.4.6-25.el8 appstream 709 k 2023-06-23T01:17:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel x86_64 2.32.1-42.el8 baseos 99 k 2023-06-23T01:17:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel x86_64 1:1.1.1k-9.el8 baseos 3.2 M 2023-06-23T01:17:07.044 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T01:17:07.045 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T01:17:07.045 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T01:17:07.045 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T01:17:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T01:17:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:17:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T01:17:07.046 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel x86_64 5.0.0-12.el8 baseos 74 k 2023-06-23T01:17:07.047 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrading: 2023-06-23T01:17:07.047 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-initscripts-updown noarch 1:1.40.16-7.el8 baseos 143 k 2023-06-23T01:17:07.048 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-debuginfod-client x86_64 0.189-2.el8 baseos 76 k 2023-06-23T01:17:07.048 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf x86_64 0.189-2.el8 baseos 232 k 2023-06-23T01:17:07.048 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libs x86_64 0.189-2.el8 baseos 302 k 2023-06-23T01:17:07.049 INFO:teuthology.orchestra.run.smithi057.stdout: iproute x86_64 6.2.0-2.el8 baseos 880 k 2023-06-23T01:17:07.049 INFO:teuthology.orchestra.run.smithi057.stdout: libblkid x86_64 2.32.1-42.el8 baseos 220 k 2023-06-23T01:17:07.049 INFO:teuthology.orchestra.run.smithi057.stdout: libfdisk x86_64 2.32.1-42.el8 baseos 253 k 2023-06-23T01:17:07.050 INFO:teuthology.orchestra.run.smithi057.stdout: libmount x86_64 2.32.1-42.el8 baseos 236 k 2023-06-23T01:17:07.050 INFO:teuthology.orchestra.run.smithi057.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:17:07.051 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T01:17:07.051 INFO:teuthology.orchestra.run.smithi057.stdout: libsmartcols x86_64 2.32.1-42.el8 baseos 179 k 2023-06-23T01:17:07.051 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid x86_64 2.32.1-42.el8 baseos 98 k 2023-06-23T01:17:07.051 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux x86_64 2.32.1-42.el8 baseos 2.6 M 2023-06-23T01:17:07.052 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux-user x86_64 2.32.1-42.el8 baseos 102 k 2023-06-23T01:17:07.052 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs x86_64 5.0.0-12.el8 baseos 1.1 M 2023-06-23T01:17:07.052 INFO:teuthology.orchestra.run.smithi057.stdout:Installing dependencies: 2023-06-23T01:17:07.052 INFO:teuthology.orchestra.run.smithi057.stdout: automake noarch 1.16.1-7.el8 appstream 713 k 2023-06-23T01:17:07.052 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T01:17:07.053 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T01:17:07.053 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T01:17:07.053 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T01:17:07.053 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T01:17:07.054 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:17:07.054 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T01:17:07.054 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T01:17:07.054 INFO:teuthology.orchestra.run.smithi057.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-06-23T01:17:07.055 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T01:17:07.055 INFO:teuthology.orchestra.run.smithi057.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-06-23T01:17:07.055 INFO:teuthology.orchestra.run.smithi057.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 baseos 48 k 2023-06-23T01:17:07.055 INFO:teuthology.orchestra.run.smithi057.stdout: krb5-devel x86_64 1.18.2-22.el8 baseos 561 k 2023-06-23T01:17:07.055 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T01:17:07.055 INFO:teuthology.orchestra.run.smithi057.stdout: libcom_err-devel x86_64 1.45.6-5.el8 baseos 39 k 2023-06-23T01:17:07.055 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran x86_64 8.5.0-19.el8 baseos 645 k 2023-06-23T01:17:07.056 INFO:teuthology.orchestra.run.smithi057.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T01:17:07.056 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath x86_64 8.5.0-19.el8 baseos 172 k 2023-06-23T01:17:07.056 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq x86_64 0.9.0-3.el8 baseos 47 k 2023-06-23T01:17:07.056 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T01:17:07.057 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-06-23T01:17:07.057 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T01:17:07.057 INFO:teuthology.orchestra.run.smithi057.stdout: libselinux-devel x86_64 2.9-8.el8 baseos 200 k 2023-06-23T01:17:07.057 INFO:teuthology.orchestra.run.smithi057.stdout: libsepol-devel x86_64 2.9-3.el8 baseos 87 k 2023-06-23T01:17:07.057 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T01:17:07.057 INFO:teuthology.orchestra.run.smithi057.stdout: libverto-devel x86_64 0.3.2-2.el8 baseos 18 k 2023-06-23T01:17:07.057 INFO:teuthology.orchestra.run.smithi057.stdout: libzstd-devel x86_64 1.4.4-1.el8 baseos 44 k 2023-06-23T01:17:07.057 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-06-23T01:17:07.058 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-06-23T01:17:07.058 INFO:teuthology.orchestra.run.smithi057.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-06-23T01:17:07.058 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-06-23T01:17:07.059 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-devel x86_64 10.32-3.el8 baseos 605 k 2023-06-23T01:17:07.059 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-utf16 x86_64 10.32-3.el8 baseos 229 k 2023-06-23T01:17:07.059 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-utf32 x86_64 10.32-3.el8 baseos 220 k 2023-06-23T01:17:07.059 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T01:17:07.059 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-06-23T01:17:07.059 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T01:17:07.060 INFO:teuthology.orchestra.run.smithi057.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T01:17:07.060 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T01:17:07.060 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T01:17:07.060 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T01:17:07.060 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T01:17:07.060 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-06-23T01:17:07.060 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T01:17:07.061 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T01:17:07.061 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-06-23T01:17:07.061 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T01:17:07.061 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T01:17:07.061 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T01:17:07.061 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-06-23T01:17:07.061 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-06-23T01:17:07.062 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-06-23T01:17:07.062 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-06-23T01:17:07.062 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T01:17:07.062 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T01:17:07.062 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-06-23T01:17:07.062 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-06-23T01:17:07.062 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T01:17:07.062 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T01:17:07.063 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-06-23T01:17:07.063 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-06-23T01:17:07.063 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-06-23T01:17:07.063 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T01:17:07.063 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T01:17:07.063 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-06-23T01:17:07.063 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-06-23T01:17:07.064 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-06-23T01:17:07.064 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-06-23T01:17:07.064 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-06-23T01:17:07.064 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz noarch 2017.2-9.el8 appstream 54 k 2023-06-23T01:17:07.064 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T01:17:07.065 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-06-23T01:17:07.065 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T01:17:07.065 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T01:17:07.065 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-06-23T01:17:07.065 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T01:17:07.065 INFO:teuthology.orchestra.run.smithi057.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T01:17:07.065 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T01:17:07.066 INFO:teuthology.orchestra.run.smithi057.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T01:17:07.066 INFO:teuthology.orchestra.run.smithi057.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T01:17:07.066 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T01:17:07.066 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T01:17:07.066 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T01:17:07.066 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-06-23T01:17:07.066 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T01:17:07.067 INFO:teuthology.orchestra.run.smithi057.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-06-23T01:17:07.067 INFO:teuthology.orchestra.run.smithi057.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T01:17:07.067 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-06-23T01:17:07.067 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel x86_64 1.2.11-21.el8 baseos 58 k 2023-06-23T01:17:07.067 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:17:07.067 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction Summary 2023-06-23T01:17:07.067 INFO:teuthology.orchestra.run.smithi057.stdout:==================================================================================================== 2023-06-23T01:17:07.068 INFO:teuthology.orchestra.run.smithi057.stdout:Install 123 Packages 2023-06-23T01:17:07.068 INFO:teuthology.orchestra.run.smithi057.stdout:Upgrade 15 Packages 2023-06-23T01:17:07.068 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:17:07.069 INFO:teuthology.orchestra.run.smithi057.stdout:Total download size: 209 M 2023-06-23T01:17:07.069 INFO:teuthology.orchestra.run.smithi057.stdout:Downloading Packages: 2023-06-23T01:17:09.768 INFO:teuthology.orchestra.run.smithi057.stdout:(1/138): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 49 kB/s | 6.4 kB 00:00 2023-06-23T01:17:10.186 INFO:teuthology.orchestra.run.smithi057.stdout:(2/138): ceph-base-17.2.6-719.g42ddc46e.el8.x86 9.4 MB/s | 5.2 MB 00:00 2023-06-23T01:17:10.219 INFO:teuthology.orchestra.run.smithi057.stdout:(3/138): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.9 MB/s | 873 kB 00:00 2023-06-23T01:17:10.244 INFO:teuthology.orchestra.run.smithi057.stdout:(4/138): ceph-immutable-object-cache-17.2.6-719 2.7 MB/s | 162 kB 00:00 2023-06-23T01:17:10.403 INFO:teuthology.orchestra.run.smithi057.stdout:(5/138): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 9.9 MB/s | 1.6 MB 00:00 2023-06-23T01:17:10.687 INFO:teuthology.orchestra.run.smithi057.stdout:(6/138): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 14 MB/s | 3.9 MB 00:00 2023-06-23T01:17:10.979 INFO:teuthology.orchestra.run.smithi057.stdout:(7/138): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 3.0 MB/s | 2.3 MB 00:00 2023-06-23T01:17:11.762 INFO:teuthology.orchestra.run.smithi057.stdout:(8/138): ceph-osd-17.2.6-719.g42ddc46e.el8.x86_ 16 MB/s | 17 MB 00:01 2023-06-23T01:17:11.804 INFO:teuthology.orchestra.run.smithi057.stdout:(9/138): ceph-selinux-17.2.6-719.g42ddc46e.el8. 586 kB/s | 24 kB 00:00 2023-06-23T01:17:12.671 INFO:teuthology.orchestra.run.smithi057.stdout:(10/138): ceph-common-17.2.6-719.g42ddc46e.el8. 7.7 MB/s | 23 MB 00:03 2023-06-23T01:17:12.780 INFO:teuthology.orchestra.run.smithi057.stdout:(11/138): cephfs-mirror-17.2.6-719.g42ddc46e.el 2.3 MB/s | 255 kB 00:00 2023-06-23T01:17:12.864 INFO:teuthology.orchestra.run.smithi057.stdout:(12/138): libcephfs-devel-17.2.6-719.g42ddc46e. 298 kB/s | 25 kB 00:00 2023-06-23T01:17:13.039 INFO:teuthology.orchestra.run.smithi057.stdout:(13/138): libcephfs2-17.2.6-719.g42ddc46e.el8.x 4.4 MB/s | 794 kB 00:00 2023-06-23T01:17:13.123 INFO:teuthology.orchestra.run.smithi057.stdout:(14/138): libcephsqlite-17.2.6-719.g42ddc46e.el 2.0 MB/s | 174 kB 00:00 2023-06-23T01:17:13.206 INFO:teuthology.orchestra.run.smithi057.stdout:(15/138): librados-devel-17.2.6-719.g42ddc46e.e 1.5 MB/s | 130 kB 00:00 2023-06-23T01:17:13.373 INFO:teuthology.orchestra.run.smithi057.stdout:(16/138): libradosstriper1-17.2.6-719.g42ddc46e 3.0 MB/s | 506 kB 00:00 2023-06-23T01:17:14.370 INFO:teuthology.orchestra.run.smithi057.stdout:(17/138): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 5.6 MB/s | 5.5 MB 00:00 2023-06-23T01:17:14.465 INFO:teuthology.orchestra.run.smithi057.stdout:(18/138): python3-ceph-argparse-17.2.6-719.g42d 469 kB/s | 45 kB 00:00 2023-06-23T01:17:14.932 INFO:teuthology.orchestra.run.smithi057.stdout:(19/138): ceph-test-17.2.6-719.g42ddc46e.el8.x8 17 MB/s | 52 MB 00:03 2023-06-23T01:17:14.958 INFO:teuthology.orchestra.run.smithi057.stdout:(20/138): python3-ceph-common-17.2.6-719.g42ddc 234 kB/s | 115 kB 00:00 2023-06-23T01:17:14.983 INFO:teuthology.orchestra.run.smithi057.stdout:(21/138): python3-cephfs-17.2.6-719.g42ddc46e.e 3.7 MB/s | 190 kB 00:00 2023-06-23T01:17:15.184 INFO:teuthology.orchestra.run.smithi057.stdout:(22/138): ceph-radosgw-17.2.6-719.g42ddc46e.el8 4.0 MB/s | 17 MB 00:04 2023-06-23T01:17:15.217 INFO:teuthology.orchestra.run.smithi057.stdout:(23/138): python3-rados-17.2.6-719.g42ddc46e.el 1.4 MB/s | 371 kB 00:00 2023-06-23T01:17:15.242 INFO:teuthology.orchestra.run.smithi057.stdout:(24/138): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.8 MB/s | 107 kB 00:00 2023-06-23T01:17:15.268 INFO:teuthology.orchestra.run.smithi057.stdout:(25/138): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.8 MB/s | 91 kB 00:00 2023-06-23T01:17:15.293 INFO:teuthology.orchestra.run.smithi057.stdout:(26/138): python3-rbd-17.2.6-719.g42ddc46e.el8. 1.1 MB/s | 357 kB 00:00 2023-06-23T01:17:15.318 INFO:teuthology.orchestra.run.smithi057.stdout:(27/138): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 3.4 MB/s | 173 kB 00:00 2023-06-23T01:17:15.335 INFO:teuthology.orchestra.run.smithi057.stdout:(28/138): ceph-grafana-dashboards-17.2.6-719.g4 552 kB/s | 23 kB 00:00 2023-06-23T01:17:15.369 INFO:teuthology.orchestra.run.smithi057.stdout:(29/138): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 2.4 MB/s | 121 kB 00:00 2023-06-23T01:17:15.502 INFO:teuthology.orchestra.run.smithi057.stdout:(30/138): ceph-mgr-dashboard-17.2.6-719.g42ddc4 10 MB/s | 1.7 MB 00:00 2023-06-23T01:17:15.569 INFO:teuthology.orchestra.run.smithi057.stdout:(31/138): ceph-mgr-modules-core-17.2.6-719.g42d 3.5 MB/s | 238 kB 00:00 2023-06-23T01:17:15.620 INFO:teuthology.orchestra.run.smithi057.stdout:(32/138): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 947 kB/s | 47 kB 00:00 2023-06-23T01:17:15.737 INFO:teuthology.orchestra.run.smithi057.stdout:(33/138): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 7.0 MB/s | 3.5 MB 00:00 2023-06-23T01:17:15.762 INFO:teuthology.orchestra.run.smithi057.stdout:(34/138): ceph-prometheus-alerts-17.2.6-719.g42 102 kB/s | 14 kB 00:00 2023-06-23T01:17:15.796 INFO:teuthology.orchestra.run.smithi057.stdout:(35/138): ceph-volume-17.2.6-719.g42ddc46e.el8. 4.4 MB/s | 264 kB 00:00 2023-06-23T01:17:15.813 INFO:teuthology.orchestra.run.smithi057.stdout:(36/138): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.6 MB/s | 80 kB 00:00 2023-06-23T01:17:15.847 INFO:teuthology.orchestra.run.smithi057.stdout:(37/138): cephfs-top-17.2.6-719.g42ddc46e.el8.n 352 kB/s | 18 kB 00:00 2023-06-23T01:17:16.030 INFO:teuthology.orchestra.run.smithi057.stdout:(38/138): ceph-mgr-diskprediction-local-17.2.6- 11 MB/s | 7.4 MB 00:00 2023-06-23T01:17:16.114 INFO:teuthology.orchestra.run.smithi057.stdout:(39/138): automake-1.16.1-7.el8.noarch.rpm 2.3 MB/s | 713 kB 00:00 2023-06-23T01:17:16.164 INFO:teuthology.orchestra.run.smithi057.stdout:(40/138): bison-3.0.4-10.el8.x86_64.rpm 2.1 MB/s | 688 kB 00:00 2023-06-23T01:17:16.190 INFO:teuthology.orchestra.run.smithi057.stdout:(41/138): jq-1.6-7.el8.x86_64.rpm 2.6 MB/s | 202 kB 00:00 2023-06-23T01:17:16.223 INFO:teuthology.orchestra.run.smithi057.stdout:(42/138): flex-2.6.1-9.el8.x86_64.rpm 1.6 MB/s | 320 kB 00:00 2023-06-23T01:17:16.307 INFO:teuthology.orchestra.run.smithi057.stdout:(43/138): librdkafka-0.11.4-3.el8.x86_64.rpm 2.4 MB/s | 354 kB 00:00 2023-06-23T01:17:16.341 INFO:teuthology.orchestra.run.smithi057.stdout:(44/138): libtool-2.4.6-25.el8.x86_64.rpm 4.6 MB/s | 709 kB 00:00 2023-06-23T01:17:16.366 INFO:teuthology.orchestra.run.smithi057.stdout:(45/138): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.8 MB/s | 259 kB 00:00 2023-06-23T01:17:16.391 INFO:teuthology.orchestra.run.smithi057.stdout:(46/138): oniguruma-6.8.2-2.el8.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2023-06-23T01:17:16.950 INFO:teuthology.orchestra.run.smithi057.stdout:(47/138): openblas-0.3.15-4.el8.x86_64.rpm 7.8 MB/s | 4.7 MB 00:00 2023-06-23T01:17:17.042 INFO:teuthology.orchestra.run.smithi057.stdout:(48/138): openblas-threads-0.3.15-4.el8.x86_64. 7.2 MB/s | 4.9 MB 00:00 2023-06-23T01:17:17.109 INFO:teuthology.orchestra.run.smithi057.stdout:(49/138): python3-jsonpatch-1.21-2.el8.noarch.r 399 kB/s | 27 kB 00:00 2023-06-23T01:17:17.134 INFO:teuthology.orchestra.run.smithi057.stdout:(50/138): python3-jinja2-2.10.1-3.el8.noarch.rp 2.9 MB/s | 538 kB 00:00 2023-06-23T01:17:17.151 INFO:teuthology.orchestra.run.smithi057.stdout:(51/138): python3-jsonpointer-1.10-11.el8.noarc 476 kB/s | 20 kB 00:00 2023-06-23T01:17:17.185 INFO:teuthology.orchestra.run.smithi057.stdout:(52/138): python3-mako-1.0.6-14.el8.noarch.rpm 3.1 MB/s | 157 kB 00:00 2023-06-23T01:17:17.202 INFO:teuthology.orchestra.run.smithi057.stdout:(53/138): python3-markupsafe-0.23-19.el8.x86_64 770 kB/s | 39 kB 00:00 2023-06-23T01:17:17.260 INFO:teuthology.orchestra.run.smithi057.stdout:(54/138): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.8 MB/s | 225 kB 00:00 2023-06-23T01:17:17.327 INFO:teuthology.orchestra.run.smithi057.stdout:(55/138): python3-babel-2.5.1-7.el8.noarch.rpm 5.1 MB/s | 4.8 MB 00:00 2023-06-23T01:17:17.378 INFO:teuthology.orchestra.run.smithi057.stdout:(56/138): python3-prettytable-0.7.2-14.el8.noar 375 kB/s | 44 kB 00:00 2023-06-23T01:17:17.403 INFO:teuthology.orchestra.run.smithi057.stdout:(57/138): python3-pyOpenSSL-19.0.0-1.el8.noarch 1.3 MB/s | 103 kB 00:00 2023-06-23T01:17:17.428 INFO:teuthology.orchestra.run.smithi057.stdout:(58/138): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-06-23T01:17:17.454 INFO:teuthology.orchestra.run.smithi057.stdout:(59/138): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-06-23T01:17:17.479 INFO:teuthology.orchestra.run.smithi057.stdout:(60/138): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-06-23T01:17:17.562 INFO:teuthology.orchestra.run.smithi057.stdout:(61/138): python3-werkzeug-0.12.2-4.el8.noarch. 5.4 MB/s | 457 kB 00:00 2023-06-23T01:17:17.663 INFO:teuthology.orchestra.run.smithi057.stdout:(62/138): socat-1.7.4.1-1.el8.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2023-06-23T01:17:17.715 INFO:teuthology.orchestra.run.smithi057.stdout:(63/138): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.3 MB/s | 71 kB 00:00 2023-06-23T01:17:17.799 INFO:teuthology.orchestra.run.smithi057.stdout:(64/138): python3-numpy-1.14.3-10.el8.x86_64.rp 6.1 MB/s | 3.7 MB 00:00 2023-06-23T01:17:17.841 INFO:teuthology.orchestra.run.smithi057.stdout:(65/138): elfutils-libelf-devel-0.189-2.el8.x86 491 kB/s | 61 kB 00:00 2023-06-23T01:17:17.924 INFO:teuthology.orchestra.run.smithi057.stdout:(66/138): keyutils-libs-devel-1.5.10-9.el8.x86_ 577 kB/s | 48 kB 00:00 2023-06-23T01:17:17.950 INFO:teuthology.orchestra.run.smithi057.stdout:(67/138): fuse-2.9.7-17.el8.x86_64.rpm 552 kB/s | 83 kB 00:00 2023-06-23T01:17:18.000 INFO:teuthology.orchestra.run.smithi057.stdout:(68/138): libaio-devel-0.3.112-1.el8.x86_64.rpm 373 kB/s | 19 kB 00:00 2023-06-23T01:17:18.050 INFO:teuthology.orchestra.run.smithi057.stdout:(69/138): libcom_err-devel-1.45.6-5.el8.x86_64. 775 kB/s | 39 kB 00:00 2023-06-23T01:17:19.272 INFO:teuthology.orchestra.run.smithi057.stdout:(70/138): python3-scipy-1.0.0-21.module_el8.5.0 7.5 MB/s | 14 MB 00:01 2023-06-23T01:17:19.342 INFO:teuthology.orchestra.run.smithi057.stdout:(71/138): krb5-devel-1.18.2-22.el8.x86_64.rpm 395 kB/s | 561 kB 00:01 2023-06-23T01:17:19.426 INFO:teuthology.orchestra.run.smithi057.stdout:(72/138): librabbitmq-0.9.0-3.el8.x86_64.rpm 563 kB/s | 47 kB 00:00 2023-06-23T01:17:19.493 INFO:teuthology.orchestra.run.smithi057.stdout:(73/138): libquadmath-8.5.0-19.el8.x86_64.rpm 781 kB/s | 172 kB 00:00 2023-06-23T01:17:19.585 INFO:teuthology.orchestra.run.smithi057.stdout:(74/138): libsepol-devel-2.9-3.el8.x86_64.rpm 953 kB/s | 87 kB 00:00 2023-06-23T01:17:19.735 INFO:teuthology.orchestra.run.smithi057.stdout:(75/138): libselinux-devel-2.9-8.el8.x86_64.rpm 648 kB/s | 200 kB 00:00 2023-06-23T01:17:19.777 INFO:teuthology.orchestra.run.smithi057.stdout:(76/138): libverto-devel-0.3.2-2.el8.x86_64.rpm 435 kB/s | 18 kB 00:00 2023-06-23T01:17:19.844 INFO:teuthology.orchestra.run.smithi057.stdout:(77/138): libgfortran-8.5.0-19.el8.x86_64.rpm 360 kB/s | 645 kB 00:01 2023-06-23T01:17:19.886 INFO:teuthology.orchestra.run.smithi057.stdout:(78/138): libzstd-devel-1.4.4-1.el8.x86_64.rpm 404 kB/s | 44 kB 00:00 2023-06-23T01:17:20.136 INFO:teuthology.orchestra.run.smithi057.stdout:(79/138): libuuid-devel-2.32.1-42.el8.x86_64.rp 180 kB/s | 99 kB 00:00 2023-06-23T01:17:20.286 INFO:teuthology.orchestra.run.smithi057.stdout:(80/138): pcre2-devel-10.32-3.el8.x86_64.rpm 1.5 MB/s | 605 kB 00:00 2023-06-23T01:17:20.320 INFO:teuthology.orchestra.run.smithi057.stdout:(81/138): pcre2-utf16-10.32-3.el8.x86_64.rpm 1.2 MB/s | 229 kB 00:00 2023-06-23T01:17:20.428 INFO:teuthology.orchestra.run.smithi057.stdout:(82/138): pcre2-utf32-10.32-3.el8.x86_64.rpm 1.5 MB/s | 220 kB 00:00 2023-06-23T01:17:20.479 INFO:teuthology.orchestra.run.smithi057.stdout:(83/138): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.5 MB/s | 249 kB 00:00 2023-06-23T01:17:20.520 INFO:teuthology.orchestra.run.smithi057.stdout:(84/138): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-06-23T01:17:20.629 INFO:teuthology.orchestra.run.smithi057.stdout:(85/138): python3-oauthlib-2.1.0-1.el8.noarch.r 1.4 MB/s | 155 kB 00:00 2023-06-23T01:17:20.721 INFO:teuthology.orchestra.run.smithi057.stdout:(86/138): python3-pycparser-2.14-14.el8.noarch. 1.2 MB/s | 109 kB 00:00 2023-06-23T01:17:20.746 INFO:teuthology.orchestra.run.smithi057.stdout:(87/138): python3-cryptography-3.2.1-6.el8.x86_ 2.0 MB/s | 641 kB 00:00 2023-06-23T01:17:20.763 INFO:teuthology.orchestra.run.smithi057.stdout:(88/138): python3-requests-oauthlib-1.0.0-1.el8 1.0 MB/s | 43 kB 00:00 2023-06-23T01:17:20.805 INFO:teuthology.orchestra.run.smithi057.stdout:(89/138): xfsprogs-devel-5.0.0-12.el8.x86_64.rp 1.2 MB/s | 74 kB 00:00 2023-06-23T01:17:20.822 INFO:teuthology.orchestra.run.smithi057.stdout:(90/138): zlib-devel-1.2.11-21.el8.x86_64.rpm 993 kB/s | 58 kB 00:00 2023-06-23T01:17:20.838 INFO:teuthology.orchestra.run.smithi057.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:20.843 INFO:teuthology.orchestra.run.smithi057.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:20.843 INFO:teuthology.orchestra.run.smithi057.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:20.845 INFO:teuthology.orchestra.run.smithi057.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:20.947 INFO:teuthology.orchestra.run.smithi057.stdout:(91/138): liboath-2.6.2-3.el8.x86_64.rpm 470 kB/s | 59 kB 00:00 2023-06-23T01:17:20.949 INFO:teuthology.orchestra.run.smithi057.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:20.950 INFO:teuthology.orchestra.run.smithi057.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:17:20.989 INFO:teuthology.orchestra.run.smithi057.stdout:(92/138): gperftools-libs-2.7-9.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2023-06-23T01:17:21.014 INFO:teuthology.orchestra.run.smithi057.stdout:(93/138): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-06-23T01:17:21.015 INFO:teuthology.orchestra.run.smithi057.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T01:17:21.056 INFO:teuthology.orchestra.run.smithi057.stdout:(94/138): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 1.0 MB/s | 44 kB 00:00 2023-06-23T01:17:21.099 INFO:teuthology.orchestra.run.smithi057.stdout:(95/138): python3-asyncssh-2.7.0-2.el8.noarch.r 4.3 MB/s | 476 kB 00:00 2023-06-23T01:17:21.124 INFO:teuthology.orchestra.run.smithi057.stdout:(96/138): python3-beautifulsoup4-4.6.3-2.el8.1. 2.7 MB/s | 185 kB 00:00 2023-06-23T01:17:21.149 INFO:teuthology.orchestra.run.smithi057.stdout:(97/138): python3-cachetools-3.1.1-4.el8.noarch 659 kB/s | 33 kB 00:00 2023-06-23T01:17:21.174 INFO:teuthology.orchestra.run.smithi057.stdout:(98/138): python3-certifi-2018.10.15-7.el8.noar 324 kB/s | 16 kB 00:00 2023-06-23T01:17:21.222 INFO:teuthology.orchestra.run.smithi057.stdout:(99/138): python3-cheroot-8.5.2-1.el8.noarch.rp 2.3 MB/s | 173 kB 00:00 2023-06-23T01:17:21.258 INFO:teuthology.orchestra.run.smithi057.stdout:(100/138): python3-cherrypy-18.4.0-1.el8.noarch 4.5 MB/s | 384 kB 00:00 2023-06-23T01:17:21.309 INFO:teuthology.orchestra.run.smithi057.stdout:(101/138): openssl-devel-1.1.1k-9.el8.x86_64.rp 2.2 MB/s | 3.2 MB 00:01 2023-06-23T01:17:21.334 INFO:teuthology.orchestra.run.smithi057.stdout:(102/138): python3-jaraco-6.2-6.el8.noarch.rpm 147 kB/s | 11 kB 00:00 2023-06-23T01:17:21.359 INFO:teuthology.orchestra.run.smithi057.stdout:(103/138): python3-google-auth-1.1.1-10.el8.noa 592 kB/s | 81 kB 00:00 2023-06-23T01:17:21.384 INFO:teuthology.orchestra.run.smithi057.stdout:(104/138): python3-jaraco-functools-2.0-4.el8.n 233 kB/s | 18 kB 00:00 2023-06-23T01:17:21.426 INFO:teuthology.orchestra.run.smithi057.stdout:(105/138): python3-more-itertools-7.2.0-3.el8.n 1.4 MB/s | 59 kB 00:00 2023-06-23T01:17:21.451 INFO:teuthology.orchestra.run.smithi057.stdout:(106/138): python3-logutils-0.3.5-11.el8.noarch 733 kB/s | 49 kB 00:00 2023-06-23T01:17:21.500 INFO:teuthology.orchestra.run.smithi057.stdout:(107/138): python3-natsort-7.1.1-2.el8.noarch.r 825 kB/s | 60 kB 00:00 2023-06-23T01:17:21.535 INFO:teuthology.orchestra.run.smithi057.stdout:(108/138): python3-pecan-1.3.2-9.el8.noarch.rpm 3.3 MB/s | 283 kB 00:00 2023-06-23T01:17:21.561 INFO:teuthology.orchestra.run.smithi057.stdout:(109/138): python3-portend-2.6-1.el8.noarch.rpm 258 kB/s | 16 kB 00:00 2023-06-23T01:17:21.586 INFO:teuthology.orchestra.run.smithi057.stdout:(110/138): python3-repoze-lru-0.7-6.el8.noarch. 671 kB/s | 34 kB 00:00 2023-06-23T01:17:21.619 INFO:teuthology.orchestra.run.smithi057.stdout:(111/138): python3-routes-2.4.1-12.el8.noarch.r 3.3 MB/s | 196 kB 00:00 2023-06-23T01:17:21.645 INFO:teuthology.orchestra.run.smithi057.stdout:(112/138): python3-rsa-4.8-1.el8.noarch.rpm 1.0 MB/s | 61 kB 00:00 2023-06-23T01:17:21.670 INFO:teuthology.orchestra.run.smithi057.stdout:(113/138): python3-simplegeneric-0.8.1-17.el8.n 374 kB/s | 19 kB 00:00 2023-06-23T01:17:21.695 INFO:teuthology.orchestra.run.smithi057.stdout:(114/138): python3-singledispatch-3.4.0.3-18.el 477 kB/s | 24 kB 00:00 2023-06-23T01:17:21.720 INFO:teuthology.orchestra.run.smithi057.stdout:(115/138): python3-tempora-1.14.1-5.el8.noarch. 583 kB/s | 29 kB 00:00 2023-06-23T01:17:21.746 INFO:teuthology.orchestra.run.smithi057.stdout:(116/138): python3-trustme-0.6.0-4.el8.noarch.r 550 kB/s | 27 kB 00:00 2023-06-23T01:17:21.804 INFO:teuthology.orchestra.run.smithi057.stdout:(117/138): python3-waitress-1.4.3-1.el8.noarch. 3.0 MB/s | 255 kB 00:00 2023-06-23T01:17:21.838 INFO:teuthology.orchestra.run.smithi057.stdout:(118/138): python3-kubernetes-11.0.0-6.el8.noar 2.3 MB/s | 1.2 MB 00:00 2023-06-23T01:17:21.863 INFO:teuthology.orchestra.run.smithi057.stdout:(119/138): python3-webob-1.8.5-1.el8.1.noarch.r 2.1 MB/s | 251 kB 00:00 2023-06-23T01:17:21.888 INFO:teuthology.orchestra.run.smithi057.stdout:(120/138): python3-websocket-client-0.56.0-5.el 732 kB/s | 61 kB 00:00 2023-06-23T01:17:21.914 INFO:teuthology.orchestra.run.smithi057.stdout:(121/138): python3-zc-lockfile-2.0-2.el8.noarch 457 kB/s | 23 kB 00:00 2023-06-23T01:17:21.939 INFO:teuthology.orchestra.run.smithi057.stdout:(122/138): python3-webtest-2.0.33-1.el8.noarch. 848 kB/s | 85 kB 00:00 2023-06-23T01:17:21.998 INFO:teuthology.orchestra.run.smithi057.stdout:(123/138): thrift-0.13.0-2.el8.x86_64.rpm 16 MB/s | 1.7 MB 00:00 2023-06-23T01:17:22.140 INFO:teuthology.orchestra.run.smithi057.stdout:(124/138): NetworkManager-initscripts-updown-1. 1.0 MB/s | 143 kB 00:00 2023-06-23T01:17:22.224 INFO:teuthology.orchestra.run.smithi057.stdout:(125/138): elfutils-debuginfod-client-0.189-2.e 908 kB/s | 76 kB 00:00 2023-06-23T01:17:22.365 INFO:teuthology.orchestra.run.smithi057.stdout:(126/138): elfutils-libelf-0.189-2.el8.x86_64.r 1.6 MB/s | 232 kB 00:00 2023-06-23T01:17:22.524 INFO:teuthology.orchestra.run.smithi057.stdout:(127/138): elfutils-libs-0.189-2.el8.x86_64.rpm 1.9 MB/s | 302 kB 00:00 2023-06-23T01:17:22.733 INFO:teuthology.orchestra.run.smithi057.stdout:(128/138): librados2-17.2.6-719.g42ddc46e.el8.x 4.2 MB/s | 3.5 MB 00:00 2023-06-23T01:17:22.799 INFO:teuthology.orchestra.run.smithi057.stdout:(129/138): librbd1-17.2.6-719.g42ddc46e.el8.x86 4.2 MB/s | 3.6 MB 00:00 2023-06-23T01:17:22.908 INFO:teuthology.orchestra.run.smithi057.stdout:(130/138): iproute-6.2.0-2.el8.x86_64.rpm 2.2 MB/s | 880 kB 00:00 2023-06-23T01:17:22.934 INFO:teuthology.orchestra.run.smithi057.stdout:(131/138): libblkid-2.32.1-42.el8.x86_64.rpm 1.1 MB/s | 220 kB 00:00 2023-06-23T01:17:23.009 INFO:teuthology.orchestra.run.smithi057.stdout:(132/138): libfdisk-2.32.1-42.el8.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2023-06-23T01:17:23.059 INFO:teuthology.orchestra.run.smithi057.stdout:(133/138): libmount-2.32.1-42.el8.x86_64.rpm 1.5 MB/s | 236 kB 00:00 2023-06-23T01:17:23.084 INFO:teuthology.orchestra.run.smithi057.stdout:(134/138): libsmartcols-2.32.1-42.el8.x86_64.rp 1.2 MB/s | 179 kB 00:00 2023-06-23T01:17:23.101 INFO:teuthology.orchestra.run.smithi057.stdout:(135/138): libuuid-2.32.1-42.el8.x86_64.rpm 1.0 MB/s | 98 kB 00:00 2023-06-23T01:17:23.176 INFO:teuthology.orchestra.run.smithi057.stdout:(136/138): util-linux-user-2.32.1-42.el8.x86_64 1.1 MB/s | 102 kB 00:00 2023-06-23T01:17:23.501 INFO:teuthology.orchestra.run.smithi057.stdout:(137/138): xfsprogs-5.0.0-12.el8.x86_64.rpm 2.7 MB/s | 1.1 MB 00:00 2023-06-23T01:17:23.956 INFO:teuthology.orchestra.run.smithi057.stdout:(138/138): util-linux-2.32.1-42.el8.x86_64.rpm 2.9 MB/s | 2.6 MB 00:00 2023-06-23T01:17:23.960 INFO:teuthology.orchestra.run.smithi057.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:17:23.962 INFO:teuthology.orchestra.run.smithi057.stdout:Total 12 MB/s | 209 MB 00:16 2023-06-23T01:17:24.809 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction check 2023-06-23T01:17:25.264 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction check succeeded. 2023-06-23T01:17:25.264 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction test 2023-06-23T01:17:26.727 INFO:teuthology.orchestra.run.smithi057.stdout:Transaction test succeeded. 2023-06-23T01:17:26.730 INFO:teuthology.orchestra.run.smithi057.stdout:Running transaction 2023-06-23T01:17:32.626 INFO:teuthology.orchestra.run.smithi057.stdout: Preparing : 1/1 2023-06-23T01:17:32.666 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libuuid-2.32.1-42.el8.x86_64 1/1 2023-06-23T01:17:32.847 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libuuid-2.32.1-42.el8.x86_64 1/153 2023-06-23T01:17:33.043 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libuuid-2.32.1-42.el8.x86_64 1/153 2023-06-23T01:17:33.246 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libblkid-2.32.1-42.el8.x86_64 2/153 2023-06-23T01:17:33.321 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libblkid-2.32.1-42.el8.x86_64 2/153 2023-06-23T01:17:33.564 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : liboath-2.6.2-3.el8.x86_64 3/153 2023-06-23T01:17:34.007 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : elfutils-libelf-0.189-2.el8.x86_64 4/153 2023-06-23T01:17:34.268 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : thrift-0.13.0-2.el8.x86_64 5/153 2023-06-23T01:17:34.700 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 6/153 2023-06-23T01:17:34.888 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 7/153 2023-06-23T01:17:35.288 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 7/153 2023-06-23T01:17:35.505 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 8/153 2023-06-23T01:17:35.931 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 8/153 2023-06-23T01:17:36.129 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 9/153 2023-06-23T01:17:36.265 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 9/153 2023-06-23T01:17:36.454 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/153 2023-06-23T01:17:36.695 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/153 2023-06-23T01:17:36.963 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 11/153 2023-06-23T01:17:37.171 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 12/153 2023-06-23T01:17:37.742 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 12/153 2023-06-23T01:17:37.954 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 13/153 2023-06-23T01:17:38.023 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 13/153 2023-06-23T01:17:38.264 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 14/153 2023-06-23T01:17:38.512 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 15/153 2023-06-23T01:17:38.679 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/153 2023-06-23T01:17:38.829 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/153 2023-06-23T01:17:39.031 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 17/153 2023-06-23T01:17:39.254 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 17/153 2023-06-23T01:17:39.529 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : xfsprogs-5.0.0-12.el8.x86_64 18/153 2023-06-23T01:17:39.668 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: xfsprogs-5.0.0-12.el8.x86_64 18/153 2023-06-23T01:17:39.894 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 19/153 2023-06-23T01:17:40.065 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : zlib-devel-1.2.11-21.el8.x86_64 20/153 2023-06-23T01:17:40.245 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libquadmath-8.5.0-19.el8.x86_64 21/153 2023-06-23T01:17:40.385 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libquadmath-8.5.0-19.el8.x86_64 21/153 2023-06-23T01:17:40.571 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libgfortran-8.5.0-19.el8.x86_64 22/153 2023-06-23T01:17:41.206 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libgfortran-8.5.0-19.el8.x86_64 22/153 2023-06-23T01:17:41.437 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-0.3.15-4.el8.x86_64 23/153 2023-06-23T01:17:42.098 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 23/153 2023-06-23T01:17:42.420 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-06-23T01:17:43.172 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 24/153 2023-06-23T01:17:43.504 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 25/153 2023-06-23T01:17:43.677 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 26/153 2023-06-23T01:17:43.951 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 27/153 2023-06-23T01:17:44.179 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/153 2023-06-23T01:17:44.386 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 29/153 2023-06-23T01:17:44.541 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 30/153 2023-06-23T01:17:44.803 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 31/153 2023-06-23T01:17:45.849 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 32/153 2023-06-23T01:17:46.194 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 33/153 2023-06-23T01:17:46.506 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 34/153 2023-06-23T01:17:46.703 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-06-23T01:17:48.608 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 35/153 2023-06-23T01:17:48.960 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 36/153 2023-06-23T01:17:49.275 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/153 2023-06-23T01:17:49.531 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rsa-4.8-1.el8.noarch 38/153 2023-06-23T01:17:49.773 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 39/153 2023-06-23T01:17:49.950 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 40/153 2023-06-23T01:17:50.212 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 41/153 2023-06-23T01:17:50.418 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : elfutils-libs-0.189-2.el8.x86_64 42/153 2023-06-23T01:17:50.648 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : elfutils-debuginfod-client-0.189-2.el8.x86_64 43/153 2023-06-23T01:17:50.911 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libfdisk-2.32.1-42.el8.x86_64 44/153 2023-06-23T01:17:50.977 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libfdisk-2.32.1-42.el8.x86_64 44/153 2023-06-23T01:17:51.202 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libmount-2.32.1-42.el8.x86_64 45/153 2023-06-23T01:17:51.278 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libmount-2.32.1-42.el8.x86_64 45/153 2023-06-23T01:17:51.427 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libuuid-devel-2.32.1-42.el8.x86_64 46/153 2023-06-23T01:17:51.644 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : libsmartcols-2.32.1-42.el8.x86_64 47/153 2023-06-23T01:17:52.234 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libsmartcols-2.32.1-42.el8.x86_64 47/153 2023-06-23T01:17:52.511 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : util-linux-2.32.1-42.el8.x86_64 48/153 2023-06-23T01:17:52.526 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: util-linux-2.32.1-42.el8.x86_64 48/153 2023-06-23T01:17:52.715 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 49/153 2023-06-23T01:17:52.993 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 50/153 2023-06-23T01:17:53.251 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 51/153 2023-06-23T01:17:53.457 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 52/153 2023-06-23T01:17:53.606 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 53/153 2023-06-23T01:17:53.825 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 54/153 2023-06-23T01:17:54.060 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 55/153 2023-06-23T01:17:54.250 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 56/153 2023-06-23T01:17:54.422 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 57/153 2023-06-23T01:17:54.601 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 58/153 2023-06-23T01:17:54.782 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 59/153 2023-06-23T01:17:54.983 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 60/153 2023-06-23T01:17:55.194 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-portend-2.6-1.el8.noarch 61/153 2023-06-23T01:17:55.363 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 62/153 2023-06-23T01:17:55.650 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 63/153 2023-06-23T01:17:55.889 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 64/153 2023-06-23T01:17:56.142 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 65/153 2023-06-23T01:17:56.537 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 66/153 2023-06-23T01:17:56.811 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 67/153 2023-06-23T01:17:56.963 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 68/153 2023-06-23T01:17:57.185 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 69/153 2023-06-23T01:17:57.427 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 70/153 2023-06-23T01:17:57.817 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 71/153 2023-06-23T01:17:58.098 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 72/153 2023-06-23T01:17:58.330 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 73/153 2023-06-23T01:17:58.589 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 74/153 2023-06-23T01:17:58.859 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 75/153 2023-06-23T01:17:59.986 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 76/153 2023-06-23T01:18:00.354 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 77/153 2023-06-23T01:18:00.634 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 78/153 2023-06-23T01:18:00.908 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 79/153 2023-06-23T01:18:01.218 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 80/153 2023-06-23T01:18:01.445 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 81/153 2023-06-23T01:18:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 82/153 2023-06-23T01:18:01.949 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : pcre2-utf32-10.32-3.el8.x86_64 83/153 2023-06-23T01:18:02.289 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : pcre2-utf16-10.32-3.el8.x86_64 84/153 2023-06-23T01:18:02.539 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : pcre2-devel-10.32-3.el8.x86_64 85/153 2023-06-23T01:18:02.751 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libzstd-devel-1.4.4-1.el8.x86_64 86/153 2023-06-23T01:18:02.943 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libverto-devel-0.3.2-2.el8.x86_64 87/153 2023-06-23T01:18:03.342 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 88/153 2023-06-23T01:18:03.605 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libselinux-devel-2.9-8.el8.x86_64 89/153 2023-06-23T01:18:03.758 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcom_err-devel-1.45.6-5.el8.x86_64 90/153 2023-06-23T01:18:04.001 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 91/153 2023-06-23T01:18:04.237 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : krb5-devel-1.18.2-22.el8.x86_64 92/153 2023-06-23T01:18:04.506 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : fuse-2.9.7-17.el8.x86_64 93/153 2023-06-23T01:18:04.777 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 94/153 2023-06-23T01:18:04.985 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 95/153 2023-06-23T01:18:05.134 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 96/153 2023-06-23T01:18:08.872 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 97/153 2023-06-23T01:18:09.116 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 97/153 2023-06-23T01:18:09.744 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 97/153 2023-06-23T01:18:10.066 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 98/153 2023-06-23T01:18:10.205 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 98/153 2023-06-23T01:18:10.391 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 99/153 2023-06-23T01:18:29.894 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 99/153 2023-06-23T01:18:29.894 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /sys 2023-06-23T01:18:29.894 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /proc 2023-06-23T01:18:29.895 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /dev 2023-06-23T01:18:29.895 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /run 2023-06-23T01:18:29.895 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /mnt 2023-06-23T01:18:29.895 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /var/tmp 2023-06-23T01:18:29.895 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /home 2023-06-23T01:18:29.895 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /tmp 2023-06-23T01:18:29.896 INFO:teuthology.orchestra.run.smithi057.stdout:skipping the directory /dev 2023-06-23T01:18:29.896 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:18:30.280 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 100/153 2023-06-23T01:18:31.842 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 100/153 2023-06-23T01:18:32.054 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 101/153 2023-06-23T01:18:32.287 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 101/153 2023-06-23T01:18:32.505 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 102/153 2023-06-23T01:18:32.918 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 102/153 2023-06-23T01:18:33.113 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 103/153 2023-06-23T01:18:33.138 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 103/153 2023-06-23T01:18:33.377 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 104/153 2023-06-23T01:18:33.618 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 105/153 2023-06-23T01:18:33.829 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 106/153 2023-06-23T01:18:33.909 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 106/153 2023-06-23T01:18:34.219 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : jq-1.6-7.el8.x86_64 107/153 2023-06-23T01:18:34.471 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : automake-1.16.1-7.el8.noarch 108/153 2023-06-23T01:18:35.034 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 109/153 2023-06-23T01:18:35.248 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 109/153 2023-06-23T01:18:35.387 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 110/153 2023-06-23T01:18:35.638 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 111/153 2023-06-23T01:18:35.878 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 112/153 2023-06-23T01:18:36.108 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 112/153 2023-06-23T01:18:36.353 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 113/153 2023-06-23T01:18:37.381 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 113/153 2023-06-23T01:18:37.621 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 114/153 2023-06-23T01:18:37.719 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 114/153 2023-06-23T01:18:38.115 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 115/153 2023-06-23T01:18:38.353 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 116/153 2023-06-23T01:18:38.391 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 116/153 2023-06-23T01:18:38.603 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 117/153 2023-06-23T01:18:38.615 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 117/153 2023-06-23T01:18:38.803 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 118/153 2023-06-23T01:18:38.953 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/153 2023-06-23T01:18:44.171 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/153 2023-06-23T01:18:46.175 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 120/153 2023-06-23T01:18:46.402 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 121/153 2023-06-23T01:18:46.479 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 121/153 2023-06-23T01:18:46.678 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 122/153 2023-06-23T01:18:46.722 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 122/153 2023-06-23T01:18:46.936 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 123/153 2023-06-23T01:18:47.306 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 123/153 2023-06-23T01:18:47.477 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 124/153 2023-06-23T01:18:47.576 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 124/153 2023-06-23T01:18:48.507 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 125/153 2023-06-23T01:18:48.840 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : openssl-devel-1:1.1.1k-9.el8.x86_64 126/153 2023-06-23T01:18:49.100 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : elfutils-libelf-devel-0.189-2.el8.x86_64 127/153 2023-06-23T01:18:49.332 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : util-linux-user-2.32.1-42.el8.x86_64 128/153 2023-06-23T01:18:49.481 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : xfsprogs-devel-5.0.0-12.el8.x86_64 129/153 2023-06-23T01:18:49.682 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 130/153 2023-06-23T01:18:49.902 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 131/153 2023-06-23T01:18:50.069 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 132/153 2023-06-23T01:18:50.397 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 133/153 2023-06-23T01:18:50.737 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : iproute-6.2.0-2.el8.x86_64 134/153 2023-06-23T01:18:50.934 INFO:teuthology.orchestra.run.smithi057.stdout: Upgrading : NetworkManager-initscripts-updown-1:1.40.16-7.el 135/153 2023-06-23T01:18:50.955 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-7.el 135/153 2023-06-23T01:18:51.151 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 136/153 2023-06-23T01:18:51.368 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : flex-2.6.1-9.el8.x86_64 137/153 2023-06-23T01:18:51.459 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 137/153 2023-06-23T01:18:51.676 INFO:teuthology.orchestra.run.smithi057.stdout: Installing : bison-3.0.4-10.el8.x86_64 138/153 2023-06-23T01:18:51.690 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 138/153 2023-06-23T01:18:51.887 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : elfutils-debuginfod-client-0.189-1.el8.x86_64 139/153 2023-06-23T01:18:52.035 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : elfutils-libs-0.189-1.el8.x86_64 140/153 2023-06-23T01:18:52.035 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : xfsprogs-5.0.0-10.el8.x86_64 141/153 2023-06-23T01:18:52.288 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: xfsprogs-5.0.0-10.el8.x86_64 141/153 2023-06-23T01:18:52.288 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 142/153 2023-06-23T01:18:52.554 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 142/153 2023-06-23T01:18:52.560 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.40.16-4.el 143/153 2023-06-23T01:18:52.714 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : NetworkManager-initscripts-updown-1:1.40.16-4.el 143/153 2023-06-23T01:18:52.714 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 144/153 2023-06-23T01:18:52.984 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 144/153 2023-06-23T01:18:53.195 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : iproute-6.2.0-1.el8.x86_64 145/153 2023-06-23T01:18:53.362 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : util-linux-user-2.32.1-41.el8.x86_64 146/153 2023-06-23T01:18:53.595 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : util-linux-2.32.1-41.el8.x86_64 147/153 2023-06-23T01:18:53.595 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libfdisk-2.32.1-41.el8.x86_64 148/153 2023-06-23T01:18:53.832 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libfdisk-2.32.1-41.el8.x86_64 148/153 2023-06-23T01:18:53.833 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libmount-2.32.1-41.el8.x86_64 149/153 2023-06-23T01:18:54.070 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libmount-2.32.1-41.el8.x86_64 149/153 2023-06-23T01:18:54.071 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libblkid-2.32.1-41.el8.x86_64 150/153 2023-06-23T01:18:54.271 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libblkid-2.32.1-41.el8.x86_64 150/153 2023-06-23T01:18:54.271 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libuuid-2.32.1-41.el8.x86_64 151/153 2023-06-23T01:18:54.495 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libuuid-2.32.1-41.el8.x86_64 151/153 2023-06-23T01:18:54.496 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : libsmartcols-2.32.1-41.el8.x86_64 152/153 2023-06-23T01:18:54.696 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: libsmartcols-2.32.1-41.el8.x86_64 152/153 2023-06-23T01:18:54.911 INFO:teuthology.orchestra.run.smithi057.stdout: Cleanup : elfutils-libelf-0.189-1.el8.x86_64 153/153 2023-06-23T01:18:55.520 INFO:teuthology.orchestra.run.smithi057.stdout: Running scriptlet: elfutils-libelf-0.189-1.el8.x86_64 153/153 2023-06-23T01:18:55.521 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/153 2023-06-23T01:18:55.521 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/153 2023-06-23T01:18:55.521 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/153 2023-06-23T01:18:55.521 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/153 2023-06-23T01:18:55.521 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/153 2023-06-23T01:18:55.521 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/153 2023-06-23T01:18:55.521 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/153 2023-06-23T01:18:55.522 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/153 2023-06-23T01:18:55.522 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/153 2023-06-23T01:18:55.522 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/153 2023-06-23T01:18:55.522 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/153 2023-06-23T01:18:55.522 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/153 2023-06-23T01:18:55.522 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/153 2023-06-23T01:18:55.522 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/153 2023-06-23T01:18:55.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/153 2023-06-23T01:18:55.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/153 2023-06-23T01:18:55.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/153 2023-06-23T01:18:55.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/153 2023-06-23T01:18:55.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/153 2023-06-23T01:18:55.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/153 2023-06-23T01:18:55.523 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/153 2023-06-23T01:18:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/153 2023-06-23T01:18:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/153 2023-06-23T01:18:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/153 2023-06-23T01:18:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/153 2023-06-23T01:18:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/153 2023-06-23T01:18:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/153 2023-06-23T01:18:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/153 2023-06-23T01:18:55.524 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/153 2023-06-23T01:18:55.525 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/153 2023-06-23T01:18:55.536 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/153 2023-06-23T01:18:55.537 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/153 2023-06-23T01:18:55.537 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/153 2023-06-23T01:18:55.537 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/153 2023-06-23T01:18:55.538 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/153 2023-06-23T01:18:55.538 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/153 2023-06-23T01:18:55.538 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/153 2023-06-23T01:18:55.539 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/153 2023-06-23T01:18:55.539 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : automake-1.16.1-7.el8.noarch 39/153 2023-06-23T01:18:55.539 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : bison-3.0.4-10.el8.x86_64 40/153 2023-06-23T01:18:55.539 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : flex-2.6.1-9.el8.x86_64 41/153 2023-06-23T01:18:55.540 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : jq-1.6-7.el8.x86_64 42/153 2023-06-23T01:18:55.540 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 43/153 2023-06-23T01:18:55.540 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 44/153 2023-06-23T01:18:55.540 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 45/153 2023-06-23T01:18:55.540 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 46/153 2023-06-23T01:18:55.540 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 47/153 2023-06-23T01:18:55.540 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 48/153 2023-06-23T01:18:55.541 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 49/153 2023-06-23T01:18:55.541 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 50/153 2023-06-23T01:18:55.541 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 51/153 2023-06-23T01:18:55.541 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 52/153 2023-06-23T01:18:55.541 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 53/153 2023-06-23T01:18:55.541 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 54/153 2023-06-23T01:18:55.541 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 55/153 2023-06-23T01:18:55.541 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 56/153 2023-06-23T01:18:55.542 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 57/153 2023-06-23T01:18:55.542 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 58/153 2023-06-23T01:18:55.542 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 59/153 2023-06-23T01:18:55.542 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 60/153 2023-06-23T01:18:55.542 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 61/153 2023-06-23T01:18:55.542 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a2 62/153 2023-06-23T01:18:55.542 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 63/153 2023-06-23T01:18:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 64/153 2023-06-23T01:18:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 65/153 2023-06-23T01:18:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-devel-0.189-2.el8.x86_64 66/153 2023-06-23T01:18:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 67/153 2023-06-23T01:18:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 68/153 2023-06-23T01:18:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : krb5-devel-1.18.2-22.el8.x86_64 69/153 2023-06-23T01:18:55.543 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 70/153 2023-06-23T01:18:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libcom_err-devel-1.45.6-5.el8.x86_64 71/153 2023-06-23T01:18:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libgfortran-8.5.0-19.el8.x86_64 72/153 2023-06-23T01:18:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libquadmath-8.5.0-19.el8.x86_64 73/153 2023-06-23T01:18:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 74/153 2023-06-23T01:18:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libselinux-devel-2.9-8.el8.x86_64 75/153 2023-06-23T01:18:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 76/153 2023-06-23T01:18:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-devel-2.32.1-42.el8.x86_64 77/153 2023-06-23T01:18:55.544 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libverto-devel-0.3.2-2.el8.x86_64 78/153 2023-06-23T01:18:55.545 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libzstd-devel-1.4.4-1.el8.x86_64 79/153 2023-06-23T01:18:55.545 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : openssl-devel-1:1.1.1k-9.el8.x86_64 80/153 2023-06-23T01:18:55.545 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : pcre2-devel-10.32-3.el8.x86_64 81/153 2023-06-23T01:18:55.545 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : pcre2-utf16-10.32-3.el8.x86_64 82/153 2023-06-23T01:18:55.545 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : pcre2-utf32-10.32-3.el8.x86_64 83/153 2023-06-23T01:18:55.545 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 84/153 2023-06-23T01:18:55.545 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 85/153 2023-06-23T01:18:55.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 86/153 2023-06-23T01:18:55.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 87/153 2023-06-23T01:18:55.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 88/153 2023-06-23T01:18:55.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 89/153 2023-06-23T01:18:55.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-devel-5.0.0-12.el8.x86_64 90/153 2023-06-23T01:18:55.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : zlib-devel-1.2.11-21.el8.x86_64 91/153 2023-06-23T01:18:55.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 92/153 2023-06-23T01:18:55.546 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 93/153 2023-06-23T01:18:55.547 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 94/153 2023-06-23T01:18:55.547 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 95/153 2023-06-23T01:18:55.547 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 96/153 2023-06-23T01:18:55.547 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 97/153 2023-06-23T01:18:55.547 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 98/153 2023-06-23T01:18:55.547 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 99/153 2023-06-23T01:18:55.547 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 100/153 2023-06-23T01:18:55.547 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 101/153 2023-06-23T01:18:55.548 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 102/153 2023-06-23T01:18:55.548 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 103/153 2023-06-23T01:18:55.548 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 104/153 2023-06-23T01:18:55.548 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 105/153 2023-06-23T01:18:55.548 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 106/153 2023-06-23T01:18:55.548 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 107/153 2023-06-23T01:18:55.548 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 108/153 2023-06-23T01:18:55.549 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 109/153 2023-06-23T01:18:55.549 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-portend-2.6-1.el8.noarch 110/153 2023-06-23T01:18:55.549 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 111/153 2023-06-23T01:18:55.549 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 112/153 2023-06-23T01:18:55.549 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 113/153 2023-06-23T01:18:55.549 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 114/153 2023-06-23T01:18:55.549 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 115/153 2023-06-23T01:18:55.549 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 116/153 2023-06-23T01:18:55.550 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 117/153 2023-06-23T01:18:55.550 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 118/153 2023-06-23T01:18:55.550 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 119/153 2023-06-23T01:18:55.550 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 120/153 2023-06-23T01:18:55.550 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 121/153 2023-06-23T01:18:55.550 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 122/153 2023-06-23T01:18:55.550 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 123/153 2023-06-23T01:18:55.550 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 124/153 2023-06-23T01:18:55.551 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 125/153 2023-06-23T01:18:55.551 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 126/153 2023-06-23T01:18:55.551 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 127/153 2023-06-23T01:18:55.551 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-7.el 128/153 2023-06-23T01:18:55.551 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : NetworkManager-initscripts-updown-1:1.40.16-4.el 129/153 2023-06-23T01:18:55.551 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-debuginfod-client-0.189-2.el8.x86_64 130/153 2023-06-23T01:18:55.551 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-debuginfod-client-0.189-1.el8.x86_64 131/153 2023-06-23T01:18:55.551 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-0.189-2.el8.x86_64 132/153 2023-06-23T01:18:55.552 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libelf-0.189-1.el8.x86_64 133/153 2023-06-23T01:18:55.552 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libs-0.189-2.el8.x86_64 134/153 2023-06-23T01:18:55.552 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : elfutils-libs-0.189-1.el8.x86_64 135/153 2023-06-23T01:18:55.552 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : iproute-6.2.0-2.el8.x86_64 136/153 2023-06-23T01:18:55.552 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : iproute-6.2.0-1.el8.x86_64 137/153 2023-06-23T01:18:55.552 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libblkid-2.32.1-42.el8.x86_64 138/153 2023-06-23T01:18:55.552 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libblkid-2.32.1-41.el8.x86_64 139/153 2023-06-23T01:18:55.553 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libfdisk-2.32.1-42.el8.x86_64 140/153 2023-06-23T01:18:55.553 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libfdisk-2.32.1-41.el8.x86_64 141/153 2023-06-23T01:18:55.553 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libmount-2.32.1-42.el8.x86_64 142/153 2023-06-23T01:18:55.553 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libmount-2.32.1-41.el8.x86_64 143/153 2023-06-23T01:18:55.553 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libsmartcols-2.32.1-42.el8.x86_64 144/153 2023-06-23T01:18:55.553 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libsmartcols-2.32.1-41.el8.x86_64 145/153 2023-06-23T01:18:55.553 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-2.32.1-42.el8.x86_64 146/153 2023-06-23T01:18:55.554 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : libuuid-2.32.1-41.el8.x86_64 147/153 2023-06-23T01:18:55.554 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-2.32.1-42.el8.x86_64 148/153 2023-06-23T01:18:55.554 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-2.32.1-41.el8.x86_64 149/153 2023-06-23T01:18:55.554 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-user-2.32.1-42.el8.x86_64 150/153 2023-06-23T01:18:55.554 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : util-linux-user-2.32.1-41.el8.x86_64 151/153 2023-06-23T01:18:55.554 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-5.0.0-12.el8.x86_64 152/153 2023-06-23T01:18:57.758 INFO:teuthology.orchestra.run.smithi057.stdout: Verifying : xfsprogs-5.0.0-10.el8.x86_64 153/153 2023-06-23T01:18:57.758 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:18:57.758 INFO:teuthology.orchestra.run.smithi057.stdout:Upgraded: 2023-06-23T01:18:57.758 INFO:teuthology.orchestra.run.smithi057.stdout: NetworkManager-initscripts-updown-1:1.40.16-7.el8.noarch 2023-06-23T01:18:57.758 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-debuginfod-client-0.189-2.el8.x86_64 2023-06-23T01:18:57.759 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-0.189-2.el8.x86_64 2023-06-23T01:18:57.759 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libs-0.189-2.el8.x86_64 2023-06-23T01:18:57.759 INFO:teuthology.orchestra.run.smithi057.stdout: iproute-6.2.0-2.el8.x86_64 2023-06-23T01:18:57.759 INFO:teuthology.orchestra.run.smithi057.stdout: libblkid-2.32.1-42.el8.x86_64 2023-06-23T01:18:57.759 INFO:teuthology.orchestra.run.smithi057.stdout: libfdisk-2.32.1-42.el8.x86_64 2023-06-23T01:18:57.760 INFO:teuthology.orchestra.run.smithi057.stdout: libmount-2.32.1-42.el8.x86_64 2023-06-23T01:18:57.760 INFO:teuthology.orchestra.run.smithi057.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.760 INFO:teuthology.orchestra.run.smithi057.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.760 INFO:teuthology.orchestra.run.smithi057.stdout: libsmartcols-2.32.1-42.el8.x86_64 2023-06-23T01:18:57.760 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-2.32.1-42.el8.x86_64 2023-06-23T01:18:57.760 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux-2.32.1-42.el8.x86_64 2023-06-23T01:18:57.761 INFO:teuthology.orchestra.run.smithi057.stdout: util-linux-user-2.32.1-42.el8.x86_64 2023-06-23T01:18:57.761 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-5.0.0-12.el8.x86_64 2023-06-23T01:18:57.761 INFO:teuthology.orchestra.run.smithi057.stdout:Installed: 2023-06-23T01:18:57.761 INFO:teuthology.orchestra.run.smithi057.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T01:18:57.761 INFO:teuthology.orchestra.run.smithi057.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T01:18:57.761 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.761 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.761 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.762 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.762 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.762 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.762 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.762 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.762 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.762 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.762 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.763 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.763 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.763 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.763 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.763 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.763 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.763 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.764 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.764 INFO:teuthology.orchestra.run.smithi057.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.764 INFO:teuthology.orchestra.run.smithi057.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.764 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.764 INFO:teuthology.orchestra.run.smithi057.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:18:57.764 INFO:teuthology.orchestra.run.smithi057.stdout: elfutils-libelf-devel-0.189-2.el8.x86_64 2023-06-23T01:18:57.764 INFO:teuthology.orchestra.run.smithi057.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T01:18:57.765 INFO:teuthology.orchestra.run.smithi057.stdout: fuse-2.9.7-17.el8.x86_64 2023-06-23T01:18:57.765 INFO:teuthology.orchestra.run.smithi057.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T01:18:57.765 INFO:teuthology.orchestra.run.smithi057.stdout: jq-1.6-7.el8.x86_64 2023-06-23T01:18:57.765 INFO:teuthology.orchestra.run.smithi057.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-06-23T01:18:57.765 INFO:teuthology.orchestra.run.smithi057.stdout: krb5-devel-1.18.2-22.el8.x86_64 2023-06-23T01:18:57.765 INFO:teuthology.orchestra.run.smithi057.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T01:18:57.765 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.766 INFO:teuthology.orchestra.run.smithi057.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.766 INFO:teuthology.orchestra.run.smithi057.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.766 INFO:teuthology.orchestra.run.smithi057.stdout: libcom_err-devel-1.45.6-5.el8.x86_64 2023-06-23T01:18:57.766 INFO:teuthology.orchestra.run.smithi057.stdout: libgfortran-8.5.0-19.el8.x86_64 2023-06-23T01:18:57.766 INFO:teuthology.orchestra.run.smithi057.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T01:18:57.766 INFO:teuthology.orchestra.run.smithi057.stdout: libquadmath-8.5.0-19.el8.x86_64 2023-06-23T01:18:57.767 INFO:teuthology.orchestra.run.smithi057.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T01:18:57.767 INFO:teuthology.orchestra.run.smithi057.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.767 INFO:teuthology.orchestra.run.smithi057.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.767 INFO:teuthology.orchestra.run.smithi057.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-06-23T01:18:57.767 INFO:teuthology.orchestra.run.smithi057.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.768 INFO:teuthology.orchestra.run.smithi057.stdout: libselinux-devel-2.9-8.el8.x86_64 2023-06-23T01:18:57.768 INFO:teuthology.orchestra.run.smithi057.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-06-23T01:18:57.768 INFO:teuthology.orchestra.run.smithi057.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T01:18:57.768 INFO:teuthology.orchestra.run.smithi057.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T01:18:57.768 INFO:teuthology.orchestra.run.smithi057.stdout: libuuid-devel-2.32.1-42.el8.x86_64 2023-06-23T01:18:57.768 INFO:teuthology.orchestra.run.smithi057.stdout: libverto-devel-0.3.2-2.el8.x86_64 2023-06-23T01:18:57.768 INFO:teuthology.orchestra.run.smithi057.stdout: libzstd-devel-1.4.4-1.el8.x86_64 2023-06-23T01:18:57.769 INFO:teuthology.orchestra.run.smithi057.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T01:18:57.769 INFO:teuthology.orchestra.run.smithi057.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T01:18:57.769 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-0.3.15-4.el8.x86_64 2023-06-23T01:18:57.769 INFO:teuthology.orchestra.run.smithi057.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-06-23T01:18:57.769 INFO:teuthology.orchestra.run.smithi057.stdout: openssl-devel-1:1.1.1k-9.el8.x86_64 2023-06-23T01:18:57.769 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-devel-10.32-3.el8.x86_64 2023-06-23T01:18:57.769 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-utf16-10.32-3.el8.x86_64 2023-06-23T01:18:57.770 INFO:teuthology.orchestra.run.smithi057.stdout: pcre2-utf32-10.32-3.el8.x86_64 2023-06-23T01:18:57.770 INFO:teuthology.orchestra.run.smithi057.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T01:18:57.770 INFO:teuthology.orchestra.run.smithi057.stdout: python3-babel-2.5.1-7.el8.noarch 2023-06-23T01:18:57.770 INFO:teuthology.orchestra.run.smithi057.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T01:18:57.770 INFO:teuthology.orchestra.run.smithi057.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T01:18:57.770 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T01:18:57.771 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.771 INFO:teuthology.orchestra.run.smithi057.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.771 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.771 INFO:teuthology.orchestra.run.smithi057.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T01:18:57.771 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-06-23T01:18:57.771 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T01:18:57.771 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T01:18:57.772 INFO:teuthology.orchestra.run.smithi057.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-06-23T01:18:57.772 INFO:teuthology.orchestra.run.smithi057.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T01:18:57.772 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T01:18:57.772 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T01:18:57.772 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-06-23T01:18:57.772 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T01:18:57.773 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T01:18:57.773 INFO:teuthology.orchestra.run.smithi057.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T01:18:57.773 INFO:teuthology.orchestra.run.smithi057.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T01:18:57.773 INFO:teuthology.orchestra.run.smithi057.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T01:18:57.773 INFO:teuthology.orchestra.run.smithi057.stdout: python3-mako-1.0.6-14.el8.noarch 2023-06-23T01:18:57.773 INFO:teuthology.orchestra.run.smithi057.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T01:18:57.773 INFO:teuthology.orchestra.run.smithi057.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T01:18:57.774 INFO:teuthology.orchestra.run.smithi057.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T01:18:57.774 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-06-23T01:18:57.774 INFO:teuthology.orchestra.run.smithi057.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-06-23T01:18:57.774 INFO:teuthology.orchestra.run.smithi057.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T01:18:57.774 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T01:18:57.774 INFO:teuthology.orchestra.run.smithi057.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T01:18:57.774 INFO:teuthology.orchestra.run.smithi057.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T01:18:57.775 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-06-23T01:18:57.775 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T01:18:57.775 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T01:18:57.775 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pycparser-2.14-14.el8.noarch 2023-06-23T01:18:57.775 INFO:teuthology.orchestra.run.smithi057.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T01:18:57.775 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.775 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.776 INFO:teuthology.orchestra.run.smithi057.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T01:18:57.776 INFO:teuthology.orchestra.run.smithi057.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T01:18:57.776 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.776 INFO:teuthology.orchestra.run.smithi057.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T01:18:57.776 INFO:teuthology.orchestra.run.smithi057.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T01:18:57.776 INFO:teuthology.orchestra.run.smithi057.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-06-23T01:18:57.777 INFO:teuthology.orchestra.run.smithi057.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T01:18:57.777 INFO:teuthology.orchestra.run.smithi057.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T01:18:57.777 INFO:teuthology.orchestra.run.smithi057.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T01:18:57.777 INFO:teuthology.orchestra.run.smithi057.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T01:18:57.777 INFO:teuthology.orchestra.run.smithi057.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T01:18:57.777 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T01:18:57.778 INFO:teuthology.orchestra.run.smithi057.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T01:18:57.778 INFO:teuthology.orchestra.run.smithi057.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T01:18:57.778 INFO:teuthology.orchestra.run.smithi057.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T01:18:57.778 INFO:teuthology.orchestra.run.smithi057.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T01:18:57.778 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.778 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.779 INFO:teuthology.orchestra.run.smithi057.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:18:57.779 INFO:teuthology.orchestra.run.smithi057.stdout: socat-1.7.4.1-1.el8.x86_64 2023-06-23T01:18:57.779 INFO:teuthology.orchestra.run.smithi057.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T01:18:57.779 INFO:teuthology.orchestra.run.smithi057.stdout: xfsprogs-devel-5.0.0-12.el8.x86_64 2023-06-23T01:18:57.779 INFO:teuthology.orchestra.run.smithi057.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T01:18:57.780 INFO:teuthology.orchestra.run.smithi057.stdout: zlib-devel-1.2.11-21.el8.x86_64 2023-06-23T01:18:57.780 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:18:57.780 INFO:teuthology.orchestra.run.smithi057.stdout:Complete! 2023-06-23T01:18:57.946 DEBUG:teuthology.parallel:result is None 2023-06-23T01:18:57.947 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:18:58.088 DEBUG:teuthology.orchestra.run.smithi057:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T01:18:58.113 INFO:teuthology.orchestra.run.smithi057.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T01:18:58.113 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T01:18:58.114 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T01:18:58.114 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:18:58.324 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T01:18:58.359 INFO:teuthology.orchestra.run.smithi123.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T01:18:58.359 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T01:18:58.360 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T01:18:58.360 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-23T01:18:58.361 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:18:58.361 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T01:18:58.393 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:18:58.394 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T01:18:58.435 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-23T01:18:58.436 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:18:58.437 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T01:18:58.467 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T01:18:58.534 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:18:58.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T01:18:58.565 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T01:18:58.635 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-23T01:18:58.636 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:18:58.636 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T01:18:58.665 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T01:18:58.733 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:18:58.733 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T01:18:58.763 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T01:18:58.833 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-23T01:18:58.833 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:18:58.833 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T01:18:58.861 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T01:18:58.929 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:18:58.929 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T01:18:58.958 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T01:18:59.026 INFO:teuthology.run_tasks:Running task ceph... 2023-06-23T01:18:59.104 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-06-23T01:18:59.105 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 777 /var/log/ceph 2023-06-23T01:18:59.108 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /var/log/ceph 2023-06-23T01:18:59.135 INFO:tasks.ceph:Disabling ceph logrotate... 2023-06-23T01:18:59.136 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T01:18:59.177 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T01:18:59.204 INFO:tasks.ceph:Creating extra log directories... 2023-06-23T01:18:59.205 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T01:18:59.245 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T01:18:59.279 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-06-23T01:18:59.280 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-06-23T01:18:59.280 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-06-23T01:18:59.280 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:18:59.281 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T01:18:59.328 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T01:18:59.446 DEBUG:teuthology.orchestra.run.smithi057:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T01:18:59.475 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:18:59.475 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T01:18:59.493 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T01:18:59.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T01:18:59.641 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-06-23T01:18:59.641 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-06-23T01:18:59.642 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939', 'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'description': 'fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7311939', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9037, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'targets': {'smithi057.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmFKNFbg8LfVlJA1uLOjbxgiw4+0fk092Q7BVjQpvTvl6JuPfSD6R0l/eOW+NQaS56gZ/rfd94TfJ8AEa86TmkfMTja/kU9VUsjKaHQuZUxAD27+I+p7anHddByJwo4BpNBmGMdqcJjJRmsBw/m81tdx0lIvubMm/9WxPGJuAMbYrWkiKvLuSEKIgPUwPJOJDgq3+XIpD2Fbm4HWnzq/Lc546n+53S1O+yBK6o9Mg4mgcQoC0ePsEG03+J43LEuie2P5SG3d6smDBaCzuHFrBt1hK9Qg2OA/SoXDus/pnea8JslBKWziXKy5pN1oBfSssmfjVGHN4XYT/HRFprap/IjmCcYBqZUxFX+MrNF9iEqSpuBbQejFhIwZDHjP1bi7Jjt3pJi0xhpX6e212ovUA6fPqhH18m7hNY+iVuYkgYO44N1qTigooLRHg62G/vfSy1vondDQxeqhkQNHMEGk44AIbrEEDa/spGSpfJpZmNBdrmkuAvsuZa7PKh26zZ+k=', 'smithi123.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOO/uyCNxlFTHtgGYzIK80IutIj5fa7IQdy3A5VOOeCfVDWpZeDUdsZg/EgE6A5K+LT9cJR/J03iCupr5cnYRzE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestSubvolumeGroupSnapshots', 'tasks.cephfs.test_volumes.TestSubvolumeSnapshots']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '076bbebc42a14f7d568aaa78eabb0038327bcb23', 'timestamp': '2023-06-22_15:25:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-06-23T01:18:59.642 DEBUG:teuthology.orchestra.run.smithi057:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T01:18:59.644 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T01:18:59.693 DEBUG:teuthology.orchestra.run.smithi057:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T01:18:59.704 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T01:18:59.760 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:18:59.760 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:18:59.789 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:18:59.790 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_1 2023-06-23T01:18:59.848 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:18:59.849 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:18:59.849 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 6h/6d Inode: 383027 Links: 1 2023-06-23T01:18:59.849 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:18:59.849 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:18:59.849 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-06-23 01:18:55.148594385 +0000 2023-06-23T01:18:59.849 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-06-23 01:09:15.988461270 +0000 2023-06-23T01:18:59.849 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-06-23 01:09:15.988461270 +0000 2023-06-23T01:18:59.850 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2023-06-23T01:18:59.850 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:18:59.915 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-06-23T01:18:59.915 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-06-23T01:18:59.916 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000311565 s, 1.6 MB/s 2023-06-23T01:18:59.916 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:18:59.976 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_2 2023-06-23T01:19:00.034 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:19:00.035 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:19:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 6h/6d Inode: 373277 Links: 1 2023-06-23T01:19:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:19:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:19:00.035 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-06-23 01:18:55.148594385 +0000 2023-06-23T01:19:00.036 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-06-23 01:09:16.503453634 +0000 2023-06-23T01:19:00.036 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-06-23 01:09:16.503453634 +0000 2023-06-23T01:19:00.036 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2023-06-23T01:19:00.036 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:19:00.101 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-06-23T01:19:00.102 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-06-23T01:19:00.102 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000280406 s, 1.8 MB/s 2023-06-23T01:19:00.103 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:19:00.161 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_3 2023-06-23T01:19:00.220 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:19:00.220 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:19:00.220 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 6h/6d Inode: 373338 Links: 1 2023-06-23T01:19:00.221 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:19:00.221 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:19:00.221 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-06-23 01:18:55.148594385 +0000 2023-06-23T01:19:00.221 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-06-23 01:09:17.045445599 +0000 2023-06-23T01:19:00.221 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-06-23 01:09:17.045445599 +0000 2023-06-23T01:19:00.222 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2023-06-23T01:19:00.222 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:19:00.286 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-06-23T01:19:00.286 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-06-23T01:19:00.286 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000367966 s, 1.4 MB/s 2023-06-23T01:19:00.287 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:19:00.346 DEBUG:teuthology.orchestra.run.smithi057:> stat /dev/vg_nvme/lv_4 2023-06-23T01:19:00.405 INFO:teuthology.orchestra.run.smithi057.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:19:00.406 INFO:teuthology.orchestra.run.smithi057.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:19:00.406 INFO:teuthology.orchestra.run.smithi057.stdout:Device: 6h/6d Inode: 379735 Links: 1 2023-06-23T01:19:00.406 INFO:teuthology.orchestra.run.smithi057.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:19:00.406 INFO:teuthology.orchestra.run.smithi057.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:19:00.406 INFO:teuthology.orchestra.run.smithi057.stdout:Access: 2023-06-23 01:18:55.148594385 +0000 2023-06-23T01:19:00.407 INFO:teuthology.orchestra.run.smithi057.stdout:Modify: 2023-06-23 01:09:17.583437622 +0000 2023-06-23T01:19:00.407 INFO:teuthology.orchestra.run.smithi057.stdout:Change: 2023-06-23 01:09:17.583437622 +0000 2023-06-23T01:19:00.407 INFO:teuthology.orchestra.run.smithi057.stdout: Birth: - 2023-06-23T01:19:00.407 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:19:00.472 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records in 2023-06-23T01:19:00.472 INFO:teuthology.orchestra.run.smithi057.stderr:1+0 records out 2023-06-23T01:19:00.472 INFO:teuthology.orchestra.run.smithi057.stderr:512 bytes copied, 0.000636249 s, 805 kB/s 2023-06-23T01:19:00.473 DEBUG:teuthology.orchestra.run.smithi057:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:19:00.533 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:00.533 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:00.533 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:19:00.555 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:19:00.555 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_1 2023-06-23T01:19:00.616 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:19:00.616 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:19:00.617 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 384083 Links: 1 2023-06-23T01:19:00.617 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:19:00.617 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:19:00.618 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-06-23 01:12:10.027045976 +0000 2023-06-23T01:19:00.618 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-06-23 01:09:06.725826945 +0000 2023-06-23T01:19:00.618 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-06-23 01:09:06.725826945 +0000 2023-06-23T01:19:00.619 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-06-23T01:19:00.619 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:19:00.685 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-06-23T01:19:00.685 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-06-23T01:19:00.686 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.00023962 s, 2.1 MB/s 2023-06-23T01:19:00.687 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:19:00.749 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_2 2023-06-23T01:19:00.807 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:19:00.807 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:19:00.808 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 384243 Links: 1 2023-06-23T01:19:00.808 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:19:00.808 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:19:00.808 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-06-23 01:12:10.028045954 +0000 2023-06-23T01:19:00.808 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-06-23 01:09:07.230815572 +0000 2023-06-23T01:19:00.808 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-06-23 01:09:07.230815572 +0000 2023-06-23T01:19:00.808 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-06-23T01:19:00.809 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:19:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-06-23T01:19:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-06-23T01:19:00.875 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000224054 s, 2.3 MB/s 2023-06-23T01:19:00.876 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:19:00.935 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_3 2023-06-23T01:19:00.994 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:19:00.994 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:19:00.994 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 384419 Links: 1 2023-06-23T01:19:00.995 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:19:00.995 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:19:00.995 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-06-23 01:12:10.028045954 +0000 2023-06-23T01:19:00.996 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-06-23 01:09:07.740804087 +0000 2023-06-23T01:19:00.996 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-06-23 01:09:07.740804087 +0000 2023-06-23T01:19:00.996 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-06-23T01:19:00.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:19:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-06-23T01:19:01.060 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-06-23T01:19:01.061 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.000287481 s, 1.8 MB/s 2023-06-23T01:19:01.061 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:19:01.120 DEBUG:teuthology.orchestra.run.smithi123:> stat /dev/vg_nvme/lv_4 2023-06-23T01:19:01.178 INFO:teuthology.orchestra.run.smithi123.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:19:01.179 INFO:teuthology.orchestra.run.smithi123.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:19:01.179 INFO:teuthology.orchestra.run.smithi123.stdout:Device: 6h/6d Inode: 384615 Links: 1 2023-06-23T01:19:01.180 INFO:teuthology.orchestra.run.smithi123.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:19:01.180 INFO:teuthology.orchestra.run.smithi123.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:19:01.180 INFO:teuthology.orchestra.run.smithi123.stdout:Access: 2023-06-23 01:12:10.028045954 +0000 2023-06-23T01:19:01.180 INFO:teuthology.orchestra.run.smithi123.stdout:Modify: 2023-06-23 01:09:08.262792331 +0000 2023-06-23T01:19:01.180 INFO:teuthology.orchestra.run.smithi123.stdout:Change: 2023-06-23 01:09:08.262792331 +0000 2023-06-23T01:19:01.180 INFO:teuthology.orchestra.run.smithi123.stdout: Birth: - 2023-06-23T01:19:01.181 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:19:01.244 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records in 2023-06-23T01:19:01.244 INFO:teuthology.orchestra.run.smithi123.stderr:1+0 records out 2023-06-23T01:19:01.244 INFO:teuthology.orchestra.run.smithi123.stderr:512 bytes copied, 0.00023471 s, 2.2 MB/s 2023-06-23T01:19:01.245 DEBUG:teuthology.orchestra.run.smithi123:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:19:01.304 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:01.304 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T01:19:01.305 INFO:tasks.ceph:Generating config... 2023-06-23T01:19:01.306 INFO:tasks.ceph:[client] client mount timeout = 600 2023-06-23T01:19:01.306 INFO:tasks.ceph:[client] debug client = 20 2023-06-23T01:19:01.307 INFO:tasks.ceph:[client] debug ms = 1 2023-06-23T01:19:01.307 INFO:tasks.ceph:[client] pid file = 2023-06-23T01:19:01.307 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-06-23T01:19:01.307 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-06-23T01:19:01.307 INFO:tasks.ceph:[mds] debug mds = 20 2023-06-23T01:19:01.308 INFO:tasks.ceph:[mds] debug ms = 1 2023-06-23T01:19:01.308 INFO:tasks.ceph:[mds] mds debug frag = True 2023-06-23T01:19:01.308 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-06-23T01:19:01.308 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-06-23T01:19:01.308 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-06-23T01:19:01.308 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-06-23T01:19:01.309 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-06-23T01:19:01.309 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-06-23T01:19:01.309 INFO:tasks.ceph:[mgr] debug client = 20 2023-06-23T01:19:01.309 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-06-23T01:19:01.309 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-06-23T01:19:01.310 INFO:tasks.ceph:[mgr] debug ms = 1 2023-06-23T01:19:01.310 INFO:tasks.ceph:[mon] debug mon = 20 2023-06-23T01:19:01.310 INFO:tasks.ceph:[mon] debug ms = 1 2023-06-23T01:19:01.310 INFO:tasks.ceph:[mon] debug paxos = 20 2023-06-23T01:19:01.310 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-06-23T01:19:01.311 INFO:tasks.ceph:[osd] bdev async discard = True 2023-06-23T01:19:01.311 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-06-23T01:19:01.311 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-06-23T01:19:01.311 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-06-23T01:19:01.311 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-06-23T01:19:01.311 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-06-23T01:19:01.312 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-06-23T01:19:01.312 INFO:tasks.ceph:[osd] debug ms = 1 2023-06-23T01:19:01.312 INFO:tasks.ceph:[osd] debug osd = 20 2023-06-23T01:19:01.312 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-06-23T01:19:01.312 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-06-23T01:19:01.313 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-06-23T01:19:01.313 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-06-23T01:19:01.313 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-06-23T01:19:01.313 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-06-23T01:19:01.313 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-06-23T01:19:01.314 INFO:tasks.ceph:Setting up mon.a... 2023-06-23T01:19:01.314 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-06-23T01:19:01.376 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.keyring 2023-06-23T01:19:01.379 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-06-23T01:19:01.426 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:19:01.456 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.57'), ('mon.b', '172.21.15.123'), ('mon.c', '[v2:172.21.15.123:3301,v1:172.21.15.123:6790]')] 2023-06-23T01:19:01.456 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.57,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-06-23T01:19:01.457 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:01.458 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T01:19:01.516 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.57 --add b 172.21.15.123 --addv c '[v2:172.21.15.123:3301,v1:172.21.15.123:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:01.604 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:01.605 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: generated fsid cbac5275-6376-4f14-97f5-8fe5f66396c8 2023-06-23T01:19:01.605 INFO:teuthology.orchestra.run.smithi057.stdout:setting min_mon_release = octopus 2023-06-23T01:19:01.605 INFO:teuthology.orchestra.run.smithi057.stdout:epoch 0 2023-06-23T01:19:01.605 INFO:teuthology.orchestra.run.smithi057.stdout:fsid cbac5275-6376-4f14-97f5-8fe5f66396c8 2023-06-23T01:19:01.605 INFO:teuthology.orchestra.run.smithi057.stdout:last_changed 2023-06-23T01:19:01.602852+0000 2023-06-23T01:19:01.606 INFO:teuthology.orchestra.run.smithi057.stdout:created 2023-06-23T01:19:01.602852+0000 2023-06-23T01:19:01.606 INFO:teuthology.orchestra.run.smithi057.stdout:min_mon_release 15 (octopus) 2023-06-23T01:19:01.606 INFO:teuthology.orchestra.run.smithi057.stdout:election_strategy: 1 2023-06-23T01:19:01.606 INFO:teuthology.orchestra.run.smithi057.stdout:0: [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] mon.a 2023-06-23T01:19:01.606 INFO:teuthology.orchestra.run.smithi057.stdout:1: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.b 2023-06-23T01:19:01.606 INFO:teuthology.orchestra.run.smithi057.stdout:2: [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] mon.c 2023-06-23T01:19:01.606 INFO:teuthology.orchestra.run.smithi057.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-06-23T01:19:01.607 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T01:19:01.626 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cbac5275-6376-4f14-97f5-8fe5f66396c8... 2023-06-23T01:19:01.628 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T01:19:01.669 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T01:19:01.713 INFO:teuthology.orchestra.run.smithi057.stdout:[global] 2023-06-23T01:19:01.714 INFO:teuthology.orchestra.run.smithi057.stdout: chdir = "" 2023-06-23T01:19:01.714 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T01:19:01.714 INFO:teuthology.orchestra.run.smithi057.stdout: auth supported = cephx 2023-06-23T01:19:01.714 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.714 INFO:teuthology.orchestra.run.smithi057.stdout: filestore xattr use omap = true 2023-06-23T01:19:01.715 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.715 INFO:teuthology.orchestra.run.smithi057.stdout: mon clock drift allowed = 1.000 2023-06-23T01:19:01.715 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.715 INFO:teuthology.orchestra.run.smithi057.stdout: osd crush chooseleaf type = 0 2023-06-23T01:19:01.715 INFO:teuthology.orchestra.run.smithi057.stdout: auth debug = true 2023-06-23T01:19:01.715 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.715 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on old message = true 2023-06-23T01:19:01.715 INFO:teuthology.orchestra.run.smithi057.stdout: ms die on bug = true 2023-06-23T01:19:01.716 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.716 INFO:teuthology.orchestra.run.smithi057.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T01:19:01.716 INFO:teuthology.orchestra.run.smithi057.stdout: mon pg warn max object skew = 0 2023-06-23T01:19:01.716 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.716 INFO:teuthology.orchestra.run.smithi057.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T01:19:01.716 INFO:teuthology.orchestra.run.smithi057.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T01:19:01.716 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.717 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default size = 2 2023-06-23T01:19:01.717 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.717 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow primary affinity = true 2023-06-23T01:19:01.717 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd allow pg remap = true 2023-06-23T01:19:01.717 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on legacy crush tunables = false 2023-06-23T01:19:01.717 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on crush straw calc version zero = false 2023-06-23T01:19:01.717 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on no sortbitwise = false 2023-06-23T01:19:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on osd down out interval zero = false 2023-06-23T01:19:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: mon warn on too few osds = false 2023-06-23T01:19:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T01:19:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T01:19:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: mon_allow_pool_size_one = true 2023-06-23T01:19:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T01:19:01.718 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.719 INFO:teuthology.orchestra.run.smithi057.stdout: osd default data pool replay window = 5 2023-06-23T01:19:01.719 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.719 INFO:teuthology.orchestra.run.smithi057.stdout: mon allow pool delete = true 2023-06-23T01:19:01.719 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.719 INFO:teuthology.orchestra.run.smithi057.stdout: mon cluster log file level = debug 2023-06-23T01:19:01.719 INFO:teuthology.orchestra.run.smithi057.stdout: debug asserts on shutdown = true 2023-06-23T01:19:01.719 INFO:teuthology.orchestra.run.smithi057.stdout: mon health detail to clog = false 2023-06-23T01:19:01.719 INFO:teuthology.orchestra.run.smithi057.stdout: mon host = "172.21.15.57,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]" 2023-06-23T01:19:01.720 INFO:teuthology.orchestra.run.smithi057.stdout: fsid = cbac5275-6376-4f14-97f5-8fe5f66396c8 2023-06-23T01:19:01.720 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.720 INFO:teuthology.orchestra.run.smithi057.stdout:[osd] 2023-06-23T01:19:01.720 INFO:teuthology.orchestra.run.smithi057.stdout: osd journal size = 100 2023-06-23T01:19:01.720 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.720 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub load threshold = 5.0 2023-06-23T01:19:01.720 INFO:teuthology.orchestra.run.smithi057.stdout: osd scrub max interval = 600 2023-06-23T01:19:01.721 INFO:teuthology.orchestra.run.smithi057.stdout: osd mclock profile = high_recovery_ops 2023-06-23T01:19:01.721 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.721 INFO:teuthology.orchestra.run.smithi057.stdout: osd recover clone overlap = true 2023-06-23T01:19:01.721 INFO:teuthology.orchestra.run.smithi057.stdout: osd recovery max chunk = 1048576 2023-06-23T01:19:01.721 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.721 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug shutdown = true 2023-06-23T01:19:01.721 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug op order = true 2023-06-23T01:19:01.721 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug verify stray on activate = true 2023-06-23T01:19:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: osd open classes on start = true 2023-06-23T01:19:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug pg log writeout = true 2023-06-23T01:19:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: osd deep scrub update digest min age = 30 2023-06-23T01:19:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: osd map max advance = 10 2023-06-23T01:19:01.722 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.723 INFO:teuthology.orchestra.run.smithi057.stdout: journal zero on create = true 2023-06-23T01:19:01.723 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.723 INFO:teuthology.orchestra.run.smithi057.stdout: filestore ondisk finisher threads = 3 2023-06-23T01:19:01.723 INFO:teuthology.orchestra.run.smithi057.stdout: filestore apply finisher threads = 3 2023-06-23T01:19:01.723 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.723 INFO:teuthology.orchestra.run.smithi057.stdout: bdev debug aio = true 2023-06-23T01:19:01.723 INFO:teuthology.orchestra.run.smithi057.stdout: osd debug misdirected ops = true 2023-06-23T01:19:01.723 INFO:teuthology.orchestra.run.smithi057.stdout: bdev async discard = True 2023-06-23T01:19:01.724 INFO:teuthology.orchestra.run.smithi057.stdout: bdev enable discard = True 2023-06-23T01:19:01.724 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore allocator = bitmap 2023-06-23T01:19:01.724 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore block size = 96636764160 2023-06-23T01:19:01.724 INFO:teuthology.orchestra.run.smithi057.stdout: bluestore fsck on mount = True 2023-06-23T01:19:01.724 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluefs = 1/20 2023-06-23T01:19:01.724 INFO:teuthology.orchestra.run.smithi057.stdout: debug bluestore = 1/20 2023-06-23T01:19:01.724 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-06-23T01:19:01.725 INFO:teuthology.orchestra.run.smithi057.stdout: debug osd = 20 2023-06-23T01:19:01.725 INFO:teuthology.orchestra.run.smithi057.stdout: debug rocksdb = 4/10 2023-06-23T01:19:01.725 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T01:19:01.725 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd full ratio = 0.9 2023-06-23T01:19:01.725 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd nearfull ratio = 0.8 2023-06-23T01:19:01.725 INFO:teuthology.orchestra.run.smithi057.stdout: osd failsafe full ratio = 0.95 2023-06-23T01:19:01.725 INFO:teuthology.orchestra.run.smithi057.stdout: osd objectstore = bluestore 2023-06-23T01:19:01.726 INFO:teuthology.orchestra.run.smithi057.stdout: osd op complaint time = 180 2023-06-23T01:19:01.726 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.726 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr] 2023-06-23T01:19:01.726 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-06-23T01:19:01.726 INFO:teuthology.orchestra.run.smithi057.stdout: debug mgr = 20 2023-06-23T01:19:01.726 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-06-23T01:19:01.726 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-06-23T01:19:01.726 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:19:01.727 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:19:01.727 INFO:teuthology.orchestra.run.smithi057.stdout: mgr/telemetry/nag = false 2023-06-23T01:19:01.727 INFO:teuthology.orchestra.run.smithi057.stdout: debug client = 20 2023-06-23T01:19:01.727 INFO:teuthology.orchestra.run.smithi057.stdout: debug finisher = 20 2023-06-23T01:19:01.727 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.727 INFO:teuthology.orchestra.run.smithi057.stdout:[mon] 2023-06-23T01:19:01.727 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-06-23T01:19:01.727 INFO:teuthology.orchestra.run.smithi057.stdout: debug mon = 20 2023-06-23T01:19:01.728 INFO:teuthology.orchestra.run.smithi057.stdout: debug paxos = 20 2023-06-23T01:19:01.728 INFO:teuthology.orchestra.run.smithi057.stdout: debug auth = 20 2023-06-23T01:19:01.728 INFO:teuthology.orchestra.run.smithi057.stdout: mon data avail warn = 5 2023-06-23T01:19:01.728 INFO:teuthology.orchestra.run.smithi057.stdout: mon mgr mkfs grace = 240 2023-06-23T01:19:01.728 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:19:01.728 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd reporter subtree level = osd 2023-06-23T01:19:01.728 INFO:teuthology.orchestra.run.smithi057.stdout: mon osd prime pg temp = true 2023-06-23T01:19:01.729 INFO:teuthology.orchestra.run.smithi057.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:19:01.729 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.729 INFO:teuthology.orchestra.run.smithi057.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T01:19:01.729 INFO:teuthology.orchestra.run.smithi057.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T01:19:01.729 INFO:teuthology.orchestra.run.smithi057.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T01:19:01.729 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.729 INFO:teuthology.orchestra.run.smithi057.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T01:19:01.729 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T01:19:01.730 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T01:19:01.730 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.730 INFO:teuthology.orchestra.run.smithi057.stdout: # 1m isn't quite enough 2023-06-23T01:19:01.730 INFO:teuthology.orchestra.run.smithi057.stdout: mon_down_mkfs_grace = 2m 2023-06-23T01:19:01.730 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.730 INFO:teuthology.orchestra.run.smithi057.stdout: mon_warn_on_filestore_osds = false 2023-06-23T01:19:01.730 INFO:teuthology.orchestra.run.smithi057.stdout: mon op complaint time = 120 2023-06-23T01:19:01.731 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:01.731 INFO:teuthology.orchestra.run.smithi057.stdout:[client] 2023-06-23T01:19:01.731 INFO:teuthology.orchestra.run.smithi057.stdout: rgw cache enabled = true 2023-06-23T01:19:01.731 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable ops log = true 2023-06-23T01:19:01.731 INFO:teuthology.orchestra.run.smithi057.stdout: rgw enable usage log = true 2023-06-23T01:19:01.731 INFO:teuthology.orchestra.run.smithi057.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T01:19:01.731 INFO:teuthology.orchestra.run.smithi057.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T01:19:01.731 INFO:teuthology.orchestra.run.smithi057.stdout: client mount timeout = 600 2023-06-23T01:19:01.732 INFO:teuthology.orchestra.run.smithi057.stdout: debug client = 20 2023-06-23T01:19:01.732 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-06-23T01:19:01.732 INFO:teuthology.orchestra.run.smithi057.stdout: pid file = "" 2023-06-23T01:19:01.732 INFO:teuthology.orchestra.run.smithi057.stdout: rados mon op timeout = 900 2023-06-23T01:19:01.732 INFO:teuthology.orchestra.run.smithi057.stdout: rados osd op timeout = 900 2023-06-23T01:19:01.732 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.a] 2023-06-23T01:19:01.732 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.b] 2023-06-23T01:19:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:[mon.c] 2023-06-23T01:19:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2023-06-23T01:19:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2023-06-23T01:19:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2023-06-23T01:19:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2023-06-23T01:19:01.733 INFO:teuthology.orchestra.run.smithi057.stdout:[mds] 2023-06-23T01:19:01.733 INFO:teuthology.orchestra.run.smithi057.stdout: debug mds = 20 2023-06-23T01:19:01.733 INFO:teuthology.orchestra.run.smithi057.stdout: debug ms = 1 2023-06-23T01:19:01.734 INFO:teuthology.orchestra.run.smithi057.stdout: mds debug frag = True 2023-06-23T01:19:01.734 INFO:teuthology.orchestra.run.smithi057.stdout: mds debug scatterstat = True 2023-06-23T01:19:01.734 INFO:teuthology.orchestra.run.smithi057.stdout: mds op complaint time = 180 2023-06-23T01:19:01.734 INFO:teuthology.orchestra.run.smithi057.stdout: mds verify scatter = True 2023-06-23T01:19:01.734 INFO:teuthology.orchestra.run.smithi057.stdout: osd op complaint time = 180 2023-06-23T01:19:01.734 INFO:teuthology.orchestra.run.smithi057.stdout: rados mon op timeout = 900 2023-06-23T01:19:01.734 INFO:teuthology.orchestra.run.smithi057.stdout: rados osd op timeout = 900 2023-06-23T01:19:01.735 INFO:teuthology.orchestra.run.smithi123.stdout:[global] 2023-06-23T01:19:01.735 INFO:teuthology.orchestra.run.smithi123.stdout: chdir = "" 2023-06-23T01:19:01.735 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T01:19:01.735 INFO:teuthology.orchestra.run.smithi123.stdout: auth supported = cephx 2023-06-23T01:19:01.735 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.735 INFO:teuthology.orchestra.run.smithi123.stdout: filestore xattr use omap = true 2023-06-23T01:19:01.735 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.736 INFO:teuthology.orchestra.run.smithi123.stdout: mon clock drift allowed = 1.000 2023-06-23T01:19:01.736 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.736 INFO:teuthology.orchestra.run.smithi123.stdout: osd crush chooseleaf type = 0 2023-06-23T01:19:01.736 INFO:teuthology.orchestra.run.smithi123.stdout: auth debug = true 2023-06-23T01:19:01.736 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.736 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on old message = true 2023-06-23T01:19:01.736 INFO:teuthology.orchestra.run.smithi123.stdout: ms die on bug = true 2023-06-23T01:19:01.736 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.737 INFO:teuthology.orchestra.run.smithi123.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T01:19:01.737 INFO:teuthology.orchestra.run.smithi123.stdout: mon pg warn max object skew = 0 2023-06-23T01:19:01.737 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.737 INFO:teuthology.orchestra.run.smithi123.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T01:19:01.737 INFO:teuthology.orchestra.run.smithi123.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T01:19:01.737 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.737 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default size = 2 2023-06-23T01:19:01.737 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.738 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow primary affinity = true 2023-06-23T01:19:01.738 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd allow pg remap = true 2023-06-23T01:19:01.738 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on legacy crush tunables = false 2023-06-23T01:19:01.738 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on crush straw calc version zero = false 2023-06-23T01:19:01.738 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on no sortbitwise = false 2023-06-23T01:19:01.738 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on osd down out interval zero = false 2023-06-23T01:19:01.738 INFO:teuthology.orchestra.run.smithi123.stdout: mon warn on too few osds = false 2023-06-23T01:19:01.739 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T01:19:01.739 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T01:19:01.739 INFO:teuthology.orchestra.run.smithi123.stdout: mon_allow_pool_size_one = true 2023-06-23T01:19:01.739 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.739 INFO:teuthology.orchestra.run.smithi123.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T01:19:01.739 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.739 INFO:teuthology.orchestra.run.smithi123.stdout: osd default data pool replay window = 5 2023-06-23T01:19:01.739 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.740 INFO:teuthology.orchestra.run.smithi123.stdout: mon allow pool delete = true 2023-06-23T01:19:01.740 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.740 INFO:teuthology.orchestra.run.smithi123.stdout: mon cluster log file level = debug 2023-06-23T01:19:01.740 INFO:teuthology.orchestra.run.smithi123.stdout: debug asserts on shutdown = true 2023-06-23T01:19:01.740 INFO:teuthology.orchestra.run.smithi123.stdout: mon health detail to clog = false 2023-06-23T01:19:01.740 INFO:teuthology.orchestra.run.smithi123.stdout: mon host = "172.21.15.57,172.21.15.123,[v2:172.21.15.123:3301,v1:172.21.15.123:6790]" 2023-06-23T01:19:01.740 INFO:teuthology.orchestra.run.smithi123.stdout: fsid = cbac5275-6376-4f14-97f5-8fe5f66396c8 2023-06-23T01:19:01.741 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.741 INFO:teuthology.orchestra.run.smithi123.stdout:[osd] 2023-06-23T01:19:01.741 INFO:teuthology.orchestra.run.smithi123.stdout: osd journal size = 100 2023-06-23T01:19:01.741 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.741 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub load threshold = 5.0 2023-06-23T01:19:01.741 INFO:teuthology.orchestra.run.smithi123.stdout: osd scrub max interval = 600 2023-06-23T01:19:01.741 INFO:teuthology.orchestra.run.smithi123.stdout: osd mclock profile = high_recovery_ops 2023-06-23T01:19:01.741 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.742 INFO:teuthology.orchestra.run.smithi123.stdout: osd recover clone overlap = true 2023-06-23T01:19:01.742 INFO:teuthology.orchestra.run.smithi123.stdout: osd recovery max chunk = 1048576 2023-06-23T01:19:01.742 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.742 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug shutdown = true 2023-06-23T01:19:01.742 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug op order = true 2023-06-23T01:19:01.743 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug verify stray on activate = true 2023-06-23T01:19:01.743 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.743 INFO:teuthology.orchestra.run.smithi123.stdout: osd open classes on start = true 2023-06-23T01:19:01.743 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug pg log writeout = true 2023-06-23T01:19:01.743 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.743 INFO:teuthology.orchestra.run.smithi123.stdout: osd deep scrub update digest min age = 30 2023-06-23T01:19:01.743 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.743 INFO:teuthology.orchestra.run.smithi123.stdout: osd map max advance = 10 2023-06-23T01:19:01.744 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.744 INFO:teuthology.orchestra.run.smithi123.stdout: journal zero on create = true 2023-06-23T01:19:01.744 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.744 INFO:teuthology.orchestra.run.smithi123.stdout: filestore ondisk finisher threads = 3 2023-06-23T01:19:01.744 INFO:teuthology.orchestra.run.smithi123.stdout: filestore apply finisher threads = 3 2023-06-23T01:19:01.744 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.744 INFO:teuthology.orchestra.run.smithi123.stdout: bdev debug aio = true 2023-06-23T01:19:01.744 INFO:teuthology.orchestra.run.smithi123.stdout: osd debug misdirected ops = true 2023-06-23T01:19:01.745 INFO:teuthology.orchestra.run.smithi123.stdout: bdev async discard = True 2023-06-23T01:19:01.745 INFO:teuthology.orchestra.run.smithi123.stdout: bdev enable discard = True 2023-06-23T01:19:01.745 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore allocator = bitmap 2023-06-23T01:19:01.745 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore block size = 96636764160 2023-06-23T01:19:01.745 INFO:teuthology.orchestra.run.smithi123.stdout: bluestore fsck on mount = True 2023-06-23T01:19:01.745 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluefs = 1/20 2023-06-23T01:19:01.745 INFO:teuthology.orchestra.run.smithi123.stdout: debug bluestore = 1/20 2023-06-23T01:19:01.745 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-06-23T01:19:01.746 INFO:teuthology.orchestra.run.smithi123.stdout: debug osd = 20 2023-06-23T01:19:01.746 INFO:teuthology.orchestra.run.smithi123.stdout: debug rocksdb = 4/10 2023-06-23T01:19:01.746 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T01:19:01.746 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd full ratio = 0.9 2023-06-23T01:19:01.746 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd nearfull ratio = 0.8 2023-06-23T01:19:01.746 INFO:teuthology.orchestra.run.smithi123.stdout: osd failsafe full ratio = 0.95 2023-06-23T01:19:01.746 INFO:teuthology.orchestra.run.smithi123.stdout: osd objectstore = bluestore 2023-06-23T01:19:01.747 INFO:teuthology.orchestra.run.smithi123.stdout: osd op complaint time = 180 2023-06-23T01:19:01.747 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.747 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr] 2023-06-23T01:19:01.747 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-06-23T01:19:01.747 INFO:teuthology.orchestra.run.smithi123.stdout: debug mgr = 20 2023-06-23T01:19:01.747 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2023-06-23T01:19:01.747 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2023-06-23T01:19:01.747 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:19:01.748 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:19:01.748 INFO:teuthology.orchestra.run.smithi123.stdout: mgr/telemetry/nag = false 2023-06-23T01:19:01.748 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2023-06-23T01:19:01.748 INFO:teuthology.orchestra.run.smithi123.stdout: debug finisher = 20 2023-06-23T01:19:01.748 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.748 INFO:teuthology.orchestra.run.smithi123.stdout:[mon] 2023-06-23T01:19:01.748 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-06-23T01:19:01.748 INFO:teuthology.orchestra.run.smithi123.stdout: debug mon = 20 2023-06-23T01:19:01.749 INFO:teuthology.orchestra.run.smithi123.stdout: debug paxos = 20 2023-06-23T01:19:01.749 INFO:teuthology.orchestra.run.smithi123.stdout: debug auth = 20 2023-06-23T01:19:01.749 INFO:teuthology.orchestra.run.smithi123.stdout: mon data avail warn = 5 2023-06-23T01:19:01.749 INFO:teuthology.orchestra.run.smithi123.stdout: mon mgr mkfs grace = 240 2023-06-23T01:19:01.749 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:19:01.749 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd reporter subtree level = osd 2023-06-23T01:19:01.749 INFO:teuthology.orchestra.run.smithi123.stdout: mon osd prime pg temp = true 2023-06-23T01:19:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:19:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T01:19:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T01:19:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T01:19:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T01:19:01.750 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T01:19:01.751 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T01:19:01.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.751 INFO:teuthology.orchestra.run.smithi123.stdout: # 1m isn't quite enough 2023-06-23T01:19:01.751 INFO:teuthology.orchestra.run.smithi123.stdout: mon_down_mkfs_grace = 2m 2023-06-23T01:19:01.751 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.751 INFO:teuthology.orchestra.run.smithi123.stdout: mon_warn_on_filestore_osds = false 2023-06-23T01:19:01.751 INFO:teuthology.orchestra.run.smithi123.stdout: mon op complaint time = 120 2023-06-23T01:19:01.752 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-06-23T01:19:01.755 INFO:teuthology.orchestra.run.smithi123.stdout:[client] 2023-06-23T01:19:01.756 INFO:teuthology.orchestra.run.smithi123.stdout: rgw cache enabled = true 2023-06-23T01:19:01.756 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable ops log = true 2023-06-23T01:19:01.756 INFO:teuthology.orchestra.run.smithi123.stdout: rgw enable usage log = true 2023-06-23T01:19:01.756 INFO:teuthology.orchestra.run.smithi123.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T01:19:01.756 INFO:teuthology.orchestra.run.smithi123.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T01:19:01.756 INFO:teuthology.orchestra.run.smithi123.stdout: client mount timeout = 600 2023-06-23T01:19:01.756 INFO:teuthology.orchestra.run.smithi123.stdout: debug client = 20 2023-06-23T01:19:01.757 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-06-23T01:19:01.757 INFO:teuthology.orchestra.run.smithi123.stdout: pid file = "" 2023-06-23T01:19:01.757 INFO:teuthology.orchestra.run.smithi123.stdout: rados mon op timeout = 900 2023-06-23T01:19:01.757 INFO:teuthology.orchestra.run.smithi123.stdout: rados osd op timeout = 900 2023-06-23T01:19:01.757 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.a] 2023-06-23T01:19:01.757 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.b] 2023-06-23T01:19:01.757 INFO:teuthology.orchestra.run.smithi123.stdout:[mon.c] 2023-06-23T01:19:01.758 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2023-06-23T01:19:01.758 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2023-06-23T01:19:01.758 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2023-06-23T01:19:01.758 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2023-06-23T01:19:01.758 INFO:teuthology.orchestra.run.smithi123.stdout:[mds] 2023-06-23T01:19:01.758 INFO:teuthology.orchestra.run.smithi123.stdout: debug mds = 20 2023-06-23T01:19:01.758 INFO:teuthology.orchestra.run.smithi123.stdout: debug ms = 1 2023-06-23T01:19:01.759 INFO:teuthology.orchestra.run.smithi123.stdout: mds debug frag = True 2023-06-23T01:19:01.759 INFO:teuthology.orchestra.run.smithi123.stdout: mds debug scatterstat = True 2023-06-23T01:19:01.759 INFO:teuthology.orchestra.run.smithi123.stdout: mds op complaint time = 180 2023-06-23T01:19:01.759 INFO:teuthology.orchestra.run.smithi123.stdout: mds verify scatter = True 2023-06-23T01:19:01.759 INFO:teuthology.orchestra.run.smithi123.stdout: osd op complaint time = 180 2023-06-23T01:19:01.760 INFO:teuthology.orchestra.run.smithi123.stdout: rados mon op timeout = 900 2023-06-23T01:19:01.760 INFO:teuthology.orchestra.run.smithi123.stdout: rados osd op timeout = 900 2023-06-23T01:19:01.762 INFO:tasks.ceph:Creating admin key on mon.a... 2023-06-23T01:19:01.763 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-06-23T01:19:01.857 INFO:tasks.ceph:Copying monmap to all nodes... 2023-06-23T01:19:01.858 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:01.858 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-06-23T01:19:01.877 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:01.877 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-06-23T01:19:01.935 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:19:01.936 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:01.936 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T01:19:01.937 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:19:02.012 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:02.013 DEBUG:teuthology.orchestra.run.smithi057:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:02.070 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:19:02.071 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:02.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T01:19:02.071 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:19:02.108 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:02.108 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:02.166 INFO:tasks.ceph:Setting up mon nodes... 2023-06-23T01:19:02.166 INFO:tasks.ceph:Setting up mgr nodes... 2023-06-23T01:19:02.167 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T01:19:02.219 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T01:19:02.221 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T01:19:02.282 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T01:19:02.284 INFO:tasks.ceph:Setting up mds nodes... 2023-06-23T01:19:02.284 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T01:19:02.336 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T01:19:02.338 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-06-23T01:19:02.365 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T01:19:02.457 INFO:teuthology.orchestra.run.smithi057.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T01:19:02.459 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-06-23T01:19:02.487 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T01:19:02.538 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T01:19:02.540 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-06-23T01:19:02.567 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T01:19:02.657 INFO:teuthology.orchestra.run.smithi123.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T01:19:02.659 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-06-23T01:19:02.685 INFO:tasks.ceph_client:Setting up client nodes... 2023-06-23T01:19:02.686 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-06-23T01:19:02.729 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-06-23T01:19:02.741 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-06-23T01:19:02.823 INFO:teuthology.orchestra.run.smithi057.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-06-23T01:19:02.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-06-23T01:19:02.875 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-06-23T01:19:02.887 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-06-23T01:19:02.967 INFO:teuthology.orchestra.run.smithi123.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-06-23T01:19:02.979 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-06-23T01:19:02.979 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi057.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi123.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T01:19:02.980 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-06-23T01:19:03.008 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:03.009 INFO:tasks.ceph:role: osd.0 2023-06-23T01:19:03.009 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:19:03.009 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T01:19:03.104 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:19:03.104 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:19:03.104 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:19:03.105 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-23T01:19:03.105 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:19:03.105 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:19:03.105 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:19:03.105 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:19:03.105 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:19:03.105 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:19:04.396 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-06-23T01:19:04.398 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-06-23T01:19:04.398 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-06-23T01:19:04.448 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-06-23T01:19:04.484 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-06-23T01:19:04.554 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:04.555 INFO:tasks.ceph:role: osd.1 2023-06-23T01:19:04.555 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:19:04.555 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T01:19:04.651 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:19:04.652 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:19:04.652 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:19:04.652 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-23T01:19:04.652 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:19:04.653 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:19:04.653 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:19:04.653 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:19:04.653 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:19:04.653 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:19:05.947 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-06-23T01:19:05.949 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-06-23T01:19:05.949 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-06-23T01:19:06.000 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-06-23T01:19:06.036 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-06-23T01:19:06.107 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:06.107 INFO:tasks.ceph:role: osd.2 2023-06-23T01:19:06.107 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:19:06.108 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T01:19:06.204 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:19:06.204 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:19:06.205 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:19:06.205 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-23T01:19:06.205 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:19:06.205 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:19:06.205 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:19:06.206 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:19:06.206 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:19:06.206 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:19:07.499 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-06-23T01:19:07.501 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-06-23T01:19:07.501 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-06-23T01:19:07.551 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-06-23T01:19:07.586 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-06-23T01:19:07.656 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:07.657 INFO:tasks.ceph:role: osd.3 2023-06-23T01:19:07.657 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:19:07.657 DEBUG:teuthology.orchestra.run.smithi057:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T01:19:07.756 INFO:teuthology.orchestra.run.smithi057.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:19:07.757 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:19:07.757 INFO:teuthology.orchestra.run.smithi057.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:19:07.757 INFO:teuthology.orchestra.run.smithi057.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-23T01:19:07.757 INFO:teuthology.orchestra.run.smithi057.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:19:07.757 INFO:teuthology.orchestra.run.smithi057.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:19:07.757 INFO:teuthology.orchestra.run.smithi057.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:19:07.758 INFO:teuthology.orchestra.run.smithi057.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:19:07.758 INFO:teuthology.orchestra.run.smithi057.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:19:07.758 INFO:teuthology.orchestra.run.smithi057.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:19:09.053 INFO:teuthology.orchestra.run.smithi057.stdout:Discarding blocks...Done. 2023-06-23T01:19:09.055 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi057.front.sepia.ceph.com -o noatime 2023-06-23T01:19:09.055 DEBUG:teuthology.orchestra.run.smithi057:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-06-23T01:19:09.105 DEBUG:teuthology.orchestra.run.smithi057:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-06-23T01:19:09.139 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:09.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:09.233+0000 7fc13abab540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-06-23T01:19:09.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:09.234+0000 7fc13abab540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-06-23T01:19:09.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:09.234+0000 7fc13abab540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:19:09.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:09.234+0000 7fc13abab540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:19:09.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:09.234+0000 7fc13abab540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:19:09.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:09.234+0000 7fc13abab540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-06-23T01:19:09.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:09.762+0000 7fc13abab540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:10.531 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:10.528+0000 7fc13abab540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:11.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:11.291+0000 7fc13abab540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:12.111 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-06-23T01:19:12.145 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:12.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:12.232+0000 7f02a068c540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-06-23T01:19:12.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:12.233+0000 7f02a068c540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-06-23T01:19:12.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:12.233+0000 7f02a068c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:19:12.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:12.233+0000 7f02a068c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:19:12.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:12.233+0000 7f02a068c540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:19:12.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:12.233+0000 7f02a068c540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-06-23T01:19:12.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:12.766+0000 7f02a068c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:13.529 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:13.527+0000 7f02a068c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:14.303 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:14.300+0000 7f02a068c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:15.114 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-06-23T01:19:15.147 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:15.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:15.233+0000 7f1d8d373540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-06-23T01:19:15.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:15.233+0000 7f1d8d373540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-06-23T01:19:15.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:15.233+0000 7f1d8d373540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:19:15.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:15.233+0000 7f1d8d373540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:19:15.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:15.233+0000 7f1d8d373540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:19:15.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:15.234+0000 7f1d8d373540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-06-23T01:19:15.766 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:15.763+0000 7f1d8d373540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:16.528 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:16.525+0000 7f1d8d373540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:17.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:17.292+0000 7f1d8d373540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:18.104 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-06-23T01:19:18.138 DEBUG:teuthology.orchestra.run.smithi057:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:18.228 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:18.225+0000 7fb4e1d95540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-06-23T01:19:18.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:18.226+0000 7fb4e1d95540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-06-23T01:19:18.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:18.226+0000 7fb4e1d95540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:19:18.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:18.226+0000 7fb4e1d95540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:19:18.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:18.226+0000 7fb4e1d95540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:19:18.229 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:18.226+0000 7fb4e1d95540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-06-23T01:19:18.754 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:18.751+0000 7fb4e1d95540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:19.515 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:19.512+0000 7fb4e1d95540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:20.277 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:20.274+0000 7fb4e1d95540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:21.077 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-06-23T01:19:21.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-06-23T01:19:21.144 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:21.144 INFO:tasks.ceph:role: osd.4 2023-06-23T01:19:21.144 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:19:21.144 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T01:19:21.221 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:19:21.221 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:19:21.222 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:19:21.222 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-23T01:19:21.222 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:19:21.222 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:19:21.222 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:19:21.222 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:19:21.222 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:19:21.222 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:19:21.654 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-06-23T01:19:21.656 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-06-23T01:19:21.656 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-06-23T01:19:21.840 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-06-23T01:19:21.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-06-23T01:19:21.949 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:21.950 INFO:tasks.ceph:role: osd.5 2023-06-23T01:19:21.950 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:19:21.950 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T01:19:22.028 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:19:22.028 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:19:22.028 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:19:22.028 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-23T01:19:22.029 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:19:22.029 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:19:22.029 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:19:22.029 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:19:22.029 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:19:22.029 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:19:22.465 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-06-23T01:19:22.466 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-06-23T01:19:22.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-06-23T01:19:22.582 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-06-23T01:19:22.616 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-06-23T01:19:22.684 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:22.685 INFO:tasks.ceph:role: osd.6 2023-06-23T01:19:22.685 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:19:22.685 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T01:19:22.759 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:19:22.760 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:19:22.760 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:19:22.760 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-23T01:19:22.760 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:19:22.760 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:19:22.761 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:19:22.761 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:19:22.761 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:19:22.761 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:19:23.209 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-06-23T01:19:23.211 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-06-23T01:19:23.211 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-06-23T01:19:23.384 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-06-23T01:19:23.418 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-06-23T01:19:23.486 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:19:23.487 INFO:tasks.ceph:role: osd.7 2023-06-23T01:19:23.487 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:19:23.487 DEBUG:teuthology.orchestra.run.smithi123:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T01:19:23.561 INFO:teuthology.orchestra.run.smithi123.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:19:23.562 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:19:23.562 INFO:teuthology.orchestra.run.smithi123.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:19:23.562 INFO:teuthology.orchestra.run.smithi123.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-06-23T01:19:23.562 INFO:teuthology.orchestra.run.smithi123.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:19:23.562 INFO:teuthology.orchestra.run.smithi123.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:19:23.562 INFO:teuthology.orchestra.run.smithi123.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:19:23.563 INFO:teuthology.orchestra.run.smithi123.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:19:23.563 INFO:teuthology.orchestra.run.smithi123.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:19:23.563 INFO:teuthology.orchestra.run.smithi123.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:19:24.004 INFO:teuthology.orchestra.run.smithi123.stdout:Discarding blocks...Done. 2023-06-23T01:19:24.005 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi123.front.sepia.ceph.com -o noatime 2023-06-23T01:19:24.006 DEBUG:teuthology.orchestra.run.smithi123:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-06-23T01:19:24.161 DEBUG:teuthology.orchestra.run.smithi123:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-06-23T01:19:24.198 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:24.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:24.297+0000 7f5f0a63f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-06-23T01:19:24.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:24.297+0000 7f5f0a63f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-06-23T01:19:24.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:24.297+0000 7f5f0a63f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:19:24.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:24.298+0000 7f5f0a63f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:19:24.301 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:24.298+0000 7f5f0a63f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:19:24.302 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:24.298+0000 7f5f0a63f540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-06-23T01:19:24.823 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:24.820+0000 7f5f0a63f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:25.589 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:25.586+0000 7f5f0a63f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:26.354 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:26.349+0000 7f5f0a63f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:27.146 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-06-23T01:19:27.179 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:27.270 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:27.268+0000 7ff97e568540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-06-23T01:19:27.271 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:27.268+0000 7ff97e568540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-06-23T01:19:27.271 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:27.268+0000 7ff97e568540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:19:27.271 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:27.268+0000 7ff97e568540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:19:27.271 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:27.268+0000 7ff97e568540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:19:27.272 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:27.268+0000 7ff97e568540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-06-23T01:19:27.795 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:27.793+0000 7ff97e568540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:28.561 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:28.558+0000 7ff97e568540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:29.322 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:29.320+0000 7ff97e568540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:29.643 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:19:29.646 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:19:30.124 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-06-23T01:19:30.156 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:30.245 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:30.241+0000 7fd56050a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-06-23T01:19:30.245 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:30.241+0000 7fd56050a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-06-23T01:19:30.245 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:30.241+0000 7fd56050a540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:19:30.246 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:30.241+0000 7fd56050a540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:19:30.246 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:30.241+0000 7fd56050a540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:19:30.246 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:30.242+0000 7fd56050a540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-06-23T01:19:30.766 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:30.763+0000 7fd56050a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:31.528 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:31.525+0000 7fd56050a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:32.294 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:32.291+0000 7fd56050a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:33.087 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-06-23T01:19:33.118 DEBUG:teuthology.orchestra.run.smithi123:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:33.204 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:33.200+0000 7fd41a77b540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-06-23T01:19:33.204 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:33.200+0000 7fd41a77b540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-06-23T01:19:33.204 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:33.200+0000 7fd41a77b540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:19:33.204 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:33.200+0000 7fd41a77b540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:19:33.204 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:33.200+0000 7fd41a77b540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:19:33.205 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:33.201+0000 7fd41a77b540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-06-23T01:19:33.724 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:33.722+0000 7fd41a77b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:34.485 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:34.481+0000 7fd41a77b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:35.252 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:35.248+0000 7fd41a77b540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:19:36.043 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-06-23T01:19:36.072 INFO:tasks.ceph:Reading keys from all nodes... 2023-06-23T01:19:36.073 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:36.073 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-06-23T01:19:36.108 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:36.108 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-06-23T01:19:36.180 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:36.180 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-06-23T01:19:36.252 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:36.253 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-06-23T01:19:36.323 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:36.323 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-06-23T01:19:36.392 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:36.392 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-06-23T01:19:36.461 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:36.462 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-06-23T01:19:36.530 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:36.530 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-06-23T01:19:36.561 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:36.562 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-06-23T01:19:36.631 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:36.631 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-06-23T01:19:36.700 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:36.701 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-06-23T01:19:36.773 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:36.774 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-06-23T01:19:36.844 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:36.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-06-23T01:19:36.912 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:36.913 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-06-23T01:19:36.980 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:36.981 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-06-23T01:19:37.003 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:37.003 DEBUG:teuthology.orchestra.run.smithi057:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-06-23T01:19:37.065 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:37.065 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-06-23T01:19:37.083 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:37.084 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-06-23T01:19:37.141 INFO:tasks.ceph:Adding keys to all mons... 2023-06-23T01:19:37.141 DEBUG:teuthology.orchestra.run.smithi057:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T01:19:37.144 DEBUG:teuthology.orchestra.run.smithi123:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T01:19:37.184 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.y] 2023-06-23T01:19:37.185 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85RkyIL5DBAAQXiNRoc01x4CaAEeipvtYQ== 2023-06-23T01:19:37.185 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.a] 2023-06-23T01:19:37.185 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85Rk7avpExAAfGdLgG8Vnv+0KGg7ccAtxw== 2023-06-23T01:19:37.185 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.c] 2023-06-23T01:19:37.185 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85RkGbwhGxAAYw/6OMBSz3jaAVko6oWNTQ== 2023-06-23T01:19:37.185 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.0] 2023-06-23T01:19:37.185 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAN85RkF0/6DRAAU4c2r6cSwdWLU6MSX4v3OQ== 2023-06-23T01:19:37.186 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.1] 2023-06-23T01:19:37.186 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAQ85RkaRjrDRAAeGhzMZrxMNP918O2GhPYbA== 2023-06-23T01:19:37.186 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.2] 2023-06-23T01:19:37.187 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAT85Rk/dPzDRAA+OUzb1ZZmTj17dra2Em6cg== 2023-06-23T01:19:37.187 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.3] 2023-06-23T01:19:37.187 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAW85RkDI9/DRAA+vL5vlUSQfORkm1oAaLLGg== 2023-06-23T01:19:37.187 INFO:teuthology.orchestra.run.smithi057.stdout:[mgr.x] 2023-06-23T01:19:37.187 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85RkwXW0EBAAHx9gpclK1quczavtYNA3CA== 2023-06-23T01:19:37.187 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.b] 2023-06-23T01:19:37.188 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85Rkw5f/HxAAKe3ADBQ59fKeVz5yZyH1QA== 2023-06-23T01:19:37.189 INFO:teuthology.orchestra.run.smithi057.stdout:[mds.d] 2023-06-23T01:19:37.189 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85RkxykSJxAA3v/C98LapqdwTlhLWpKBYA== 2023-06-23T01:19:37.189 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.4] 2023-06-23T01:19:37.189 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAc85RkHbTHERAAVsU9YNef/KRKaFKzTM9QdA== 2023-06-23T01:19:37.189 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.5] 2023-06-23T01:19:37.189 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAf85RknAIDEBAAOlJjFdTDwLAZYHDgaKsXDA== 2023-06-23T01:19:37.189 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.6] 2023-06-23T01:19:37.189 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAi85RkRTtrDhAA3fv7XPQQnGwgE7S9N7VGDA== 2023-06-23T01:19:37.190 INFO:teuthology.orchestra.run.smithi057.stdout:[osd.7] 2023-06-23T01:19:37.190 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAl85RktvD8CxAATW9TIcM5tnwDjtQZw7wqyg== 2023-06-23T01:19:37.190 INFO:teuthology.orchestra.run.smithi057.stdout:[client.0] 2023-06-23T01:19:37.190 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85Rk2INXKxAA8UOZ/1WCVTLTO6k3yd2lGw== 2023-06-23T01:19:37.190 INFO:teuthology.orchestra.run.smithi057.stdout:[client.2] 2023-06-23T01:19:37.190 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85RklUL2MBAA8HjijFDAjWC4Yl6+PFPCoQ== 2023-06-23T01:19:37.190 INFO:teuthology.orchestra.run.smithi057.stdout:[client.1] 2023-06-23T01:19:37.191 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85RkonwVNBAA97xneP+Vuzxc503WchGgug== 2023-06-23T01:19:37.191 INFO:teuthology.orchestra.run.smithi057.stdout:[client.3] 2023-06-23T01:19:37.191 INFO:teuthology.orchestra.run.smithi057.stdout: key = AQAG85RkrSCSORAA/GXHWWN//iPMgNvlUOhOLg== 2023-06-23T01:19:37.207 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.y] 2023-06-23T01:19:37.207 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85RkyIL5DBAAQXiNRoc01x4CaAEeipvtYQ== 2023-06-23T01:19:37.208 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.a] 2023-06-23T01:19:37.208 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85Rk7avpExAAfGdLgG8Vnv+0KGg7ccAtxw== 2023-06-23T01:19:37.208 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.c] 2023-06-23T01:19:37.208 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85RkGbwhGxAAYw/6OMBSz3jaAVko6oWNTQ== 2023-06-23T01:19:37.208 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.0] 2023-06-23T01:19:37.209 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAN85RkF0/6DRAAU4c2r6cSwdWLU6MSX4v3OQ== 2023-06-23T01:19:37.209 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.1] 2023-06-23T01:19:37.209 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAQ85RkaRjrDRAAeGhzMZrxMNP918O2GhPYbA== 2023-06-23T01:19:37.209 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.2] 2023-06-23T01:19:37.209 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAT85Rk/dPzDRAA+OUzb1ZZmTj17dra2Em6cg== 2023-06-23T01:19:37.209 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.3] 2023-06-23T01:19:37.210 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAW85RkDI9/DRAA+vL5vlUSQfORkm1oAaLLGg== 2023-06-23T01:19:37.210 INFO:teuthology.orchestra.run.smithi123.stdout:[mgr.x] 2023-06-23T01:19:37.210 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85RkwXW0EBAAHx9gpclK1quczavtYNA3CA== 2023-06-23T01:19:37.210 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.b] 2023-06-23T01:19:37.210 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85Rkw5f/HxAAKe3ADBQ59fKeVz5yZyH1QA== 2023-06-23T01:19:37.210 INFO:teuthology.orchestra.run.smithi123.stdout:[mds.d] 2023-06-23T01:19:37.210 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85RkxykSJxAA3v/C98LapqdwTlhLWpKBYA== 2023-06-23T01:19:37.210 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.4] 2023-06-23T01:19:37.211 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAc85RkHbTHERAAVsU9YNef/KRKaFKzTM9QdA== 2023-06-23T01:19:37.211 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.5] 2023-06-23T01:19:37.211 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAf85RknAIDEBAAOlJjFdTDwLAZYHDgaKsXDA== 2023-06-23T01:19:37.211 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.6] 2023-06-23T01:19:37.211 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAi85RkRTtrDhAA3fv7XPQQnGwgE7S9N7VGDA== 2023-06-23T01:19:37.211 INFO:teuthology.orchestra.run.smithi123.stdout:[osd.7] 2023-06-23T01:19:37.211 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAl85RktvD8CxAATW9TIcM5tnwDjtQZw7wqyg== 2023-06-23T01:19:37.211 INFO:teuthology.orchestra.run.smithi123.stdout:[client.0] 2023-06-23T01:19:37.212 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85Rk2INXKxAA8UOZ/1WCVTLTO6k3yd2lGw== 2023-06-23T01:19:37.212 INFO:teuthology.orchestra.run.smithi123.stdout:[client.2] 2023-06-23T01:19:37.212 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85RklUL2MBAA8HjijFDAjWC4Yl6+PFPCoQ== 2023-06-23T01:19:37.212 INFO:teuthology.orchestra.run.smithi123.stdout:[client.1] 2023-06-23T01:19:37.212 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85RkonwVNBAA97xneP+Vuzxc503WchGgug== 2023-06-23T01:19:37.212 INFO:teuthology.orchestra.run.smithi123.stdout:[client.3] 2023-06-23T01:19:37.213 INFO:teuthology.orchestra.run.smithi123.stdout: key = AQAG85RkrSCSORAA/GXHWWN//iPMgNvlUOhOLg== 2023-06-23T01:19:37.214 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:19:37.236 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:19:37.303 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:19:37.306 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:19:37.361 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:19:37.363 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:19:37.416 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.419 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.473 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.481 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.531 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.534 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.584 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.586 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.636 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:19:37.639 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:19:37.689 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:19:37.691 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:19:37.744 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:19:37.746 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:19:37.798 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.801 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.850 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.853 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.903 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.905 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.956 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:37.960 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:19:38.010 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:19:38.013 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:19:38.066 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:19:38.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:19:38.123 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:19:38.127 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:19:38.179 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:19:38.183 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:19:38.236 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-06-23T01:19:38.236 DEBUG:teuthology.orchestra.run.smithi057:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-06-23T01:19:38.265 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:19:38.362 DEBUG:teuthology.orchestra.run.smithi057:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-06-23T01:19:38.390 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-06-23T01:19:38.417 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:19:38.511 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-06-23T01:19:38.538 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-06-23T01:19:38.609 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:19:38.702 DEBUG:teuthology.orchestra.run.smithi123:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-06-23T01:19:38.730 DEBUG:teuthology.orchestra.run.smithi057:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:38.732 DEBUG:teuthology.orchestra.run.smithi123:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:19:38.787 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-06-23T01:19:38.788 INFO:tasks.ceph.mon.a:Restarting daemon 2023-06-23T01:19:38.788 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-06-23T01:19:38.790 INFO:tasks.ceph.mon.a:Started 2023-06-23T01:19:38.791 INFO:tasks.ceph.mon.b:Restarting daemon 2023-06-23T01:19:38.791 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-06-23T01:19:38.829 INFO:tasks.ceph.mon.b:Started 2023-06-23T01:19:38.830 INFO:tasks.ceph.mon.c:Restarting daemon 2023-06-23T01:19:38.830 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-06-23T01:19:38.832 INFO:tasks.ceph.mon.c:Started 2023-06-23T01:19:38.832 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-06-23T01:19:38.833 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-06-23T01:19:38.833 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-06-23T01:19:38.835 INFO:tasks.ceph.mgr.y:Started 2023-06-23T01:19:38.835 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-06-23T01:19:38.835 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-06-23T01:19:38.837 INFO:tasks.ceph.mgr.x:Started 2023-06-23T01:19:38.837 INFO:tasks.ceph:Setting crush tunables to default 2023-06-23T01:19:38.838 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd crush tunables default 2023-06-23T01:19:38.964 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.960+0000 7f5a0f605700 1 Processor -- start 2023-06-23T01:19:38.965 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.961+0000 7f5a0f605700 1 -- start start 2023-06-23T01:19:38.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.961+0000 7f5a0f605700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5a0815a230 0x7f5a080cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:38.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.961+0000 7f5a0f605700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 0x7f5a080caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:38.966 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.961+0000 7f5a0f605700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 0x7f5a080cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:38.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.961+0000 7f5a0f605700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0815af00 con 0x7f5a0815a230 2023-06-23T01:19:38.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0f605700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a0815b1e0 con 0x7f5a0815a600 2023-06-23T01:19:38.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0f605700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0815b4c0 con 0x7f5a08157620 2023-06-23T01:19:38.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0d3a1700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5a0815a230 0x7f5a080cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:38.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0d3a1700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f5a0815a230 0x7f5a080cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59170/0 (socket says 172.21.15.57:59170) 2023-06-23T01:19:38.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/388445285 learned_addr learned my addr 172.21.15.57:0/388445285 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:38.967 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0dba2700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 0x7f5a080cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:38.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0cba0700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 0x7f5a080caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:38.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/388445285 >> v2:172.21.15.57:3300/0 conn(0x7f5a0815a230 msgr2=0x7f5a080cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:19:38.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/388445285 >> v2:172.21.15.57:3300/0 conn(0x7f5a0815a230 msgr2=0x7f5a080cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:19:38.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0dba2700 1 -- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 msgr2=0x7f5a080cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-23T01:19:38.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0dba2700 1 -- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 msgr2=0x7f5a080cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:19:38.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.57:3300/0 conn(0x7f5a0815a230 0x7f5a080cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:19:38.968 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0dba2700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 0x7f5a080cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:19:38.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0dba2700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 0x7f5a080cb410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:19:38.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0cba0700 1 -- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 msgr2=0x7f5a080caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-06-23T01:19:38.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0cba0700 1 -- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 msgr2=0x7f5a080caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:19:38.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.57:3300/0 conn(0x7f5a0815a230 0x7f5a080cd3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:19:38.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0cba0700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 0x7f5a080caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:19:38.969 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:38.962+0000 7f5a0cba0700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 0x7f5a080caed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:19:39.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.57:3300/0 conn(0x7f5a0815a230 0x7f5a080cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.167 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0dba2700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 0x7f5a080cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0cba0700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 0x7f5a080caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 msgr2=0x7f5a080caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:39.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 0x7f5a080caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.168 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 msgr2=0x7f5a080cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:39.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 0x7f5a080cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/388445285 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0815b7a0 con 0x7f5a0815a230 2023-06-23T01:19:39.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0815a230 0x7f5a080cd3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a080d0fb0 tx=0x7f5a00008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b15fdfa7d52c61b2 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:39.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f59fffff700 1 -- 172.21.15.57:0/388445285 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0000c070 con 0x7f5a0815a230 2023-06-23T01:19:39.169 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f59fffff700 1 -- 172.21.15.57:0/388445285 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a00005a00 con 0x7f5a0815a230 2023-06-23T01:19:39.170 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.163+0000 7f59fffff700 1 -- 172.21.15.57:0/388445285 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00005dc0 con 0x7f5a0815a230 2023-06-23T01:19:39.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 -- 172.21.15.57:0/388445285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0815a230 msgr2=0x7f5a080cd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:39.171 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0815a230 0x7f5a080cd3f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5a080d0fb0 tx=0x7f5a00008040 comp rx=0 tx=0).stop 2023-06-23T01:19:39.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 -- 172.21.15.57:0/388445285 shutdown_connections 2023-06-23T01:19:39.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 --2- 172.21.15.57:0/388445285 >> v2:172.21.15.123:3300/0 conn(0x7f5a0815a600 0x7f5a080cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a0815a230 0x7f5a080cd3f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 --2- 172.21.15.57:0/388445285 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a08157620 0x7f5a080caed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.172 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 -- 172.21.15.57:0/388445285 >> 172.21.15.57:0/388445285 conn(0x7f5a08067340 msgr2=0x7f5a08067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:39.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 -- 172.21.15.57:0/388445285 shutdown_connections 2023-06-23T01:19:39.173 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.164+0000 7f5a0f605700 1 -- 172.21.15.57:0/388445285 wait complete. 2023-06-23T01:19:39.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.166+0000 7f5a0f605700 1 Processor -- start 2023-06-23T01:19:39.174 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.166+0000 7f5a0f605700 1 -- start start 2023-06-23T01:19:39.177 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0f605700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a08157620 0x7f5a08144d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:39.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0f605700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a0815a230 0x7f5a080afe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:39.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0f605700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a0815a600 0x7f5a080b0390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:39.178 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0f605700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a080b15d0 con 0x7f5a08157620 2023-06-23T01:19:39.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0f605700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a080c94e0 con 0x7f5a0815a600 2023-06-23T01:19:39.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0f605700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a080b09c0 con 0x7f5a0815a230 2023-06-23T01:19:39.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a08157620 0x7f5a08144d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a08157620 0x7f5a08144d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59204/0 (socket says 172.21.15.57:59204) 2023-06-23T01:19:39.179 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/3112259256 learned_addr learned my addr 172.21.15.57:0/3112259256 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:39.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0cba0700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a0815a230 0x7f5a080afe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.180 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0dba2700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a0815a600 0x7f5a080b0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/3112259256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a0815a230 msgr2=0x7f5a080afe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:39.181 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a0815a230 0x7f5a080afe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/3112259256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a0815a600 msgr2=0x7f5a080b0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:39.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a0815a600 0x7f5a080b0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.182 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 -- 172.21.15.57:0/3112259256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a000094f0 con 0x7f5a08157620 2023-06-23T01:19:39.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.168+0000 7f5a0d3a1700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a08157620 0x7f5a08144d70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5a00006f90 tx=0x7f5a00019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:39.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.169+0000 7f59fd7fa700 1 -- 172.21.15.57:0/3112259256 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00013430 con 0x7f5a08157620 2023-06-23T01:19:39.183 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.169+0000 7f59fd7fa700 1 -- 172.21.15.57:0/3112259256 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a000135b0 con 0x7f5a08157620 2023-06-23T01:19:39.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.169+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a080b0ca0 con 0x7f5a08157620 2023-06-23T01:19:39.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.169+0000 7f59fd7fa700 1 -- 172.21.15.57:0/3112259256 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00013970 con 0x7f5a08157620 2023-06-23T01:19:39.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.169+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a080b1260 con 0x7f5a08157620 2023-06-23T01:19:39.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.169+0000 7f59fd7fa700 1 -- 172.21.15.57:0/3112259256 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f5a0001e420 con 0x7f5a08157620 2023-06-23T01:19:39.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.169+0000 7f59fd7fa700 1 -- 172.21.15.57:0/3112259256 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f5a00026740 con 0x7f5a08157620 2023-06-23T01:19:39.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.170+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a08064b50 con 0x7f5a08157620 2023-06-23T01:19:39.185 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.172+0000 7f59fd7fa700 1 -- 172.21.15.57:0/3112259256 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f5a0002b060 con 0x7f5a08157620 2023-06-23T01:19:39.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.248+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5a0815ba80 con 0x7f5a08157620 2023-06-23T01:19:39.257 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.253+0000 7f59fd7fa700 1 -- 172.21.15.57:0/3112259256 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5a00035090 con 0x7f5a08157620 2023-06-23T01:19:39.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a08157620 msgr2=0x7f5a08144d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:39.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a08157620 0x7f5a08144d70 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f5a00006f90 tx=0x7f5a00019000 comp rx=0 tx=0).stop 2023-06-23T01:19:39.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 shutdown_connections 2023-06-23T01:19:39.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5a0815a600 0x7f5a080b0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f5a08157620 0x7f5a08144d70 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 --2- 172.21.15.57:0/3112259256 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f5a0815a230 0x7f5a080afe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 >> 172.21.15.57:0/3112259256 conn(0x7f5a08067340 msgr2=0x7f5a08158f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:39.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 shutdown_connections 2023-06-23T01:19:39.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:39.254+0000 7f5a0f605700 1 -- 172.21.15.57:0/3112259256 wait complete. 2023-06-23T01:19:39.260 INFO:teuthology.orchestra.run.smithi057.stderr:adjusted tunables profile to default 2023-06-23T01:19:39.272 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-06-23T01:19:39.272 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:39.272 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-06-23T01:19:39.275 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:39.272+0000 7fc2582221c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:19:39.301 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:39.302 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-06-23T01:19:39.378 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:39.378 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-06-23T01:19:39.454 DEBUG:teuthology.orchestra.run.smithi057:> set -ex 2023-06-23T01:19:39.455 DEBUG:teuthology.orchestra.run.smithi057:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-06-23T01:19:39.462 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:39.459+0000 7fc2582221c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:19:39.490 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:39.491 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-06-23T01:19:39.520 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:39.521 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-06-23T01:19:39.587 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:39.588 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-06-23T01:19:39.644 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:39.640+0000 7f8140b431c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:19:39.654 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-06-23T01:19:39.655 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-06-23T01:19:39.721 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new bb40d155-cae0-4fda-b7af-ce312c958bbf 0 2023-06-23T01:19:39.822 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:39.819+0000 7fc2582221c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:19:39.842 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:39.838+0000 7f8140b431c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:19:39.901 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.897+0000 7f9a5d889700 1 Processor -- start 2023-06-23T01:19:39.901 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.897+0000 7f9a5d889700 1 -- start start 2023-06-23T01:19:39.903 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a5d889700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580a74c0 0x7f9a580acab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:39.903 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a5d889700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9a58068a60 0x7f9a580a5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:39.904 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a5d889700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a580ad620 con 0x7f9a580acff0 2023-06-23T01:19:39.904 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a5d889700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a580b2870 con 0x7f9a58068a60 2023-06-23T01:19:39.904 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a5d889700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5815cf20 con 0x7f9a580a74c0 2023-06-23T01:19:39.904 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a56ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580a74c0 0x7f9a580acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.904 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a567fc700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f9a580acff0 0x7f9a58068350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53782/0 (socket says 172.21.15.123:53782) 2023-06-23T01:19:39.904 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a567fc700 1 -- 172.21.15.123:0/508526288 learned_addr learned my addr 172.21.15.123:0/508526288 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-06-23T01:19:39.904 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.898+0000 7f9a577fe700 1 --2- 172.21.15.123:0/508526288 >> v2:172.21.15.57:3300/0 conn(0x7f9a58068a60 0x7f9a580a5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.905 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a55ffb700 1 -- 172.21.15.123:0/508526288 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404520077 0 0) 0x7f9a580ad620 con 0x7f9a580acff0 2023-06-23T01:19:39.905 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a55ffb700 1 -- 172.21.15.123:0/508526288 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a40003570 con 0x7f9a580acff0 2023-06-23T01:19:39.905 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a56ffd700 1 -- 172.21.15.123:0/508526288 >> v2:172.21.15.57:3300/0 conn(0x7f9a58068a60 msgr2=0x7f9a580a5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:39.905 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a56ffd700 1 --2- 172.21.15.123:0/508526288 >> v2:172.21.15.57:3300/0 conn(0x7f9a58068a60 0x7f9a580a5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.905 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a56ffd700 1 -- 172.21.15.123:0/508526288 >> v1:172.21.15.123:6789/0 conn(0x7f9a580acff0 legacy=0x7f9a58068350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:39.905 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a56ffd700 1 -- 172.21.15.123:0/508526288 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a5815d200 con 0x7f9a580a74c0 2023-06-23T01:19:39.906 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a56ffd700 1 --2- 172.21.15.123:0/508526288 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580a74c0 0x7f9a580acab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9a480098e0 tx=0x7f9a480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cb3637e97c62566 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:39.906 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a55ffb700 1 -- 172.21.15.123:0/508526288 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48011630 con 0x7f9a580a74c0 2023-06-23T01:19:39.906 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a55ffb700 1 -- 172.21.15.123:0/508526288 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a48011790 con 0x7f9a580a74c0 2023-06-23T01:19:39.906 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.899+0000 7f9a55ffb700 1 -- 172.21.15.123:0/508526288 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a48011b30 con 0x7f9a580a74c0 2023-06-23T01:19:39.906 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.900+0000 7f9a5d889700 1 -- 172.21.15.123:0/508526288 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580a74c0 msgr2=0x7f9a580acab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:39.907 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.900+0000 7f9a5d889700 1 --2- 172.21.15.123:0/508526288 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580a74c0 0x7f9a580acab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9a480098e0 tx=0x7f9a480091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.907 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.900+0000 7f9a5d889700 1 -- 172.21.15.123:0/508526288 shutdown_connections 2023-06-23T01:19:39.907 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.900+0000 7f9a5d889700 1 --2- 172.21.15.123:0/508526288 >> v2:172.21.15.57:3300/0 conn(0x7f9a58068a60 0x7f9a580a5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.907 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.900+0000 7f9a5d889700 1 --2- 172.21.15.123:0/508526288 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580a74c0 0x7f9a580acab0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.907 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.900+0000 7f9a5d889700 1 -- 172.21.15.123:0/508526288 >> 172.21.15.123:0/508526288 conn(0x7f9a5806eba0 msgr2=0x7f9a5806efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:39.908 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.900+0000 7f9a5d889700 1 -- 172.21.15.123:0/508526288 shutdown_connections 2023-06-23T01:19:39.908 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.900+0000 7f9a5d889700 1 -- 172.21.15.123:0/508526288 wait complete. 2023-06-23T01:19:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.901+0000 7f9a5d889700 1 Processor -- start 2023-06-23T01:19:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.901+0000 7f9a5d889700 1 -- start start 2023-06-23T01:19:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a5d889700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a58068a60 0x7f9a581419c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a5d889700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a580a74c0 0x7f9a580c2700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a5d889700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580acff0 0x7f9a580c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:39.910 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a5d889700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a58141f00 con 0x7f9a58068a60 2023-06-23T01:19:39.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a5d889700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5815c500 con 0x7f9a580a74c0 2023-06-23T01:19:39.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a5d889700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a5815b8f0 con 0x7f9a580acff0 2023-06-23T01:19:39.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a567fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a580a74c0 0x7f9a580c2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a577fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580acff0 0x7f9a580c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a567fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a580a74c0 0x7f9a580c2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49842/0 (socket says 172.21.15.123:49842) 2023-06-23T01:19:39.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a567fc700 1 -- 172.21.15.123:0/1956051911 learned_addr learned my addr 172.21.15.123:0/1956051911 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:39.911 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.902+0000 7f9a56ffd700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a58068a60 0x7f9a581419c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:39.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a577fe700 1 -- 172.21.15.123:0/1956051911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a580a74c0 msgr2=0x7f9a580c2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:39.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a577fe700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a580a74c0 0x7f9a580c2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a577fe700 1 -- 172.21.15.123:0/1956051911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a58068a60 msgr2=0x7f9a581419c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:39.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a577fe700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a58068a60 0x7f9a581419c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:39.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a577fe700 1 -- 172.21.15.123:0/1956051911 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a48009420 con 0x7f9a580acff0 2023-06-23T01:19:39.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a567fc700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a580a74c0 0x7f9a580c2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:39.912 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a56ffd700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a58068a60 0x7f9a581419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:39.913 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a577fe700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580acff0 0x7f9a580c2c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009e20 tx=0x7f9a5000b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:39.913 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a3effd700 1 -- 172.21.15.123:0/1956051911 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000caa0 con 0x7f9a580acff0 2023-06-23T01:19:39.913 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a3effd700 1 -- 172.21.15.123:0/1956051911 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5000f040 con 0x7f9a580acff0 2023-06-23T01:19:39.913 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a5815bbd0 con 0x7f9a580acff0 2023-06-23T01:19:39.913 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a3effd700 1 -- 172.21.15.123:0/1956051911 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5000ce50 con 0x7f9a580acff0 2023-06-23T01:19:39.913 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a5815c190 con 0x7f9a580acff0 2023-06-23T01:19:39.913 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a3effd700 1 -- 172.21.15.123:0/1956051911 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9a50014690 con 0x7f9a580acff0 2023-06-23T01:19:39.914 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.903+0000 7f9a3effd700 1 -- 172.21.15.123:0/1956051911 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9a5001d880 con 0x7f9a580acff0 2023-06-23T01:19:39.914 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.904+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a58064b50 con 0x7f9a580acff0 2023-06-23T01:19:39.914 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.906+0000 7f9a3effd700 1 -- 172.21.15.123:0/1956051911 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f9a5003b6e0 con 0x7f9a580acff0 2023-06-23T01:19:39.992 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.988+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "bb40d155-cae0-4fda-b7af-ce312c958bbf", "id": 0} v 0) v1 -- 0x7f9a5815c640 con 0x7f9a580acff0 2023-06-23T01:19:39.998 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.994+0000 7f9a3effd700 1 -- 172.21.15.123:0/1956051911 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "bb40d155-cae0-4fda-b7af-ce312c958bbf", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9a5002a070 con 0x7f9a580acff0 2023-06-23T01:19:39.999 INFO:teuthology.orchestra.run.smithi123.stdout:0 2023-06-23T01:19:40.000 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580acff0 msgr2=0x7f9a580c2c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.001 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580acff0 0x7f9a580c2c40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f9a50009e20 tx=0x7f9a5000b040 comp rx=0 tx=0).stop 2023-06-23T01:19:40.001 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 shutdown_connections 2023-06-23T01:19:40.001 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a580acff0 0x7f9a580c2c40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.002 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a58068a60 0x7f9a581419c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.002 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 --2- 172.21.15.123:0/1956051911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a580a74c0 0x7f9a580c2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.002 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 >> 172.21.15.123:0/1956051911 conn(0x7f9a5806eba0 msgr2=0x7f9a580cc700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.002 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 shutdown_connections 2023-06-23T01:19:40.003 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:39.995+0000 7f9a5d889700 1 -- 172.21.15.123:0/1956051911 wait complete. 2023-06-23T01:19:40.011 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new e2dc7148-fc1a-48a4-8b06-013164077813 1 2023-06-23T01:19:40.129 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.125+0000 7f648f4ad700 1 Processor -- start 2023-06-23T01:19:40.129 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.125+0000 7f648f4ad700 1 -- start start 2023-06-23T01:19:40.131 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648f4ad700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f64880a74c0 0x7f64880acab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648f4ad700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f64880acff0 0x7f6488068350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648f4ad700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64880ad620 con 0x7f6488068890 2023-06-23T01:19:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648f4ad700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6488068c90 con 0x7f64880a74c0 2023-06-23T01:19:40.132 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648f4ad700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64880a6c10 con 0x7f64880acff0 2023-06-23T01:19:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648da4a700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f6488068890 0x7f6488070220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53784/0 (socket says 172.21.15.123:53784) 2023-06-23T01:19:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648da4a700 1 -- 172.21.15.123:0/3580072738 learned_addr learned my addr 172.21.15.123:0/3580072738 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-06-23T01:19:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648ca48700 1 --2- 172.21.15.123:0/3580072738 >> v2:172.21.15.123:3300/0 conn(0x7f64880acff0 0x7f6488068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.133 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.126+0000 7f648d249700 1 --2- 172.21.15.123:0/3580072738 >> v2:172.21.15.57:3300/0 conn(0x7f64880a74c0 0x7f64880acab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.134 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f648ca48700 1 -- 172.21.15.123:0/3580072738 >> v2:172.21.15.57:3300/0 conn(0x7f64880a74c0 msgr2=0x7f64880acab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f648ca48700 1 --2- 172.21.15.123:0/3580072738 >> v2:172.21.15.57:3300/0 conn(0x7f64880a74c0 0x7f64880acab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.135 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f648ca48700 1 -- 172.21.15.123:0/3580072738 >> v1:172.21.15.123:6789/0 conn(0x7f6488068890 legacy=0x7f6488070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f647ffff700 1 -- 172.21.15.123:0/3580072738 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661068573 0 0) 0x7f64880ad620 con 0x7f6488068890 2023-06-23T01:19:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f648ca48700 1 -- 172.21.15.123:0/3580072738 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64880b2870 con 0x7f64880acff0 2023-06-23T01:19:40.136 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f648ca48700 1 --2- 172.21.15.123:0/3580072738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880acff0 0x7f6488068350 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f648806e0a0 tx=0x7f64780091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b963b02d19fabf5c server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:40.137 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f647ffff700 1 -- 172.21.15.123:0/3580072738 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478011630 con 0x7f64880acff0 2023-06-23T01:19:40.137 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f647ffff700 1 -- 172.21.15.123:0/3580072738 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6478011790 con 0x7f64880acff0 2023-06-23T01:19:40.137 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.127+0000 7f647ffff700 1 -- 172.21.15.123:0/3580072738 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6478011b30 con 0x7f64880acff0 2023-06-23T01:19:40.138 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.128+0000 7f648f4ad700 1 -- 172.21.15.123:0/3580072738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880acff0 msgr2=0x7f6488068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.138 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.128+0000 7f648f4ad700 1 --2- 172.21.15.123:0/3580072738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880acff0 0x7f6488068350 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f648806e0a0 tx=0x7f64780091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.138 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.128+0000 7f648f4ad700 1 -- 172.21.15.123:0/3580072738 shutdown_connections 2023-06-23T01:19:40.139 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.128+0000 7f648f4ad700 1 --2- 172.21.15.123:0/3580072738 >> v2:172.21.15.57:3300/0 conn(0x7f64880a74c0 0x7f64880acab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.139 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.128+0000 7f648f4ad700 1 --2- 172.21.15.123:0/3580072738 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880acff0 0x7f6488068350 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.139 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.128+0000 7f648f4ad700 1 -- 172.21.15.123:0/3580072738 >> 172.21.15.123:0/3580072738 conn(0x7f648806eba0 msgr2=0x7f648806efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.139 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.128+0000 7f648f4ad700 1 -- 172.21.15.123:0/3580072738 shutdown_connections 2023-06-23T01:19:40.140 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.128+0000 7f648f4ad700 1 -- 172.21.15.123:0/3580072738 wait complete. 2023-06-23T01:19:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.129+0000 7f648f4ad700 1 Processor -- start 2023-06-23T01:19:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.129+0000 7f648f4ad700 1 -- start start 2023-06-23T01:19:40.144 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648f4ad700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6488068890 0x7f64880cc450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648f4ad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880a74c0 0x7f648816cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648f4ad700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64880acff0 0x7f648816d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648f4ad700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64880c3f40 con 0x7f6488068890 2023-06-23T01:19:40.145 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648f4ad700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f648816de30 con 0x7f64880a74c0 2023-06-23T01:19:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648f4ad700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64880ca1b0 con 0x7f64880acff0 2023-06-23T01:19:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648ca48700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880a74c0 0x7f648816cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648ca48700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880a74c0 0x7f648816cfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49860/0 (socket says 172.21.15.123:49860) 2023-06-23T01:19:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648da4a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64880acff0 0x7f648816d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648ca48700 1 -- 172.21.15.123:0/3030805540 learned_addr learned my addr 172.21.15.123:0/3030805540 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:40.146 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648d249700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6488068890 0x7f64880cc450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648d249700 1 -- 172.21.15.123:0/3030805540 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64880acff0 msgr2=0x7f648816d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648d249700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64880acff0 0x7f648816d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648d249700 1 -- 172.21.15.123:0/3030805540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880a74c0 msgr2=0x7f648816cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648d249700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880a74c0 0x7f648816cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648d249700 1 -- 172.21.15.123:0/3030805540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6478009420 con 0x7f6488068890 2023-06-23T01:19:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648da4a700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64880acff0 0x7f648816d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:40.147 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648ca48700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880a74c0 0x7f648816cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648d249700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6488068890 0x7f64880cc450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f648400ea50 tx=0x7f64840091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f647d7fa700 1 -- 172.21.15.123:0/3030805540 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64840095d0 con 0x7f6488068890 2023-06-23T01:19:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f647d7fa700 1 -- 172.21.15.123:0/3030805540 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f648400c7f0 con 0x7f6488068890 2023-06-23T01:19:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64880ca490 con 0x7f6488068890 2023-06-23T01:19:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.130+0000 7f647d7fa700 1 -- 172.21.15.123:0/3030805540 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f648400cbb0 con 0x7f6488068890 2023-06-23T01:19:40.148 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.131+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64880caa50 con 0x7f6488068890 2023-06-23T01:19:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.131+0000 7f647d7fa700 1 -- 172.21.15.123:0/3030805540 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f6484005c80 con 0x7f6488068890 2023-06-23T01:19:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.131+0000 7f647d7fa700 1 -- 172.21.15.123:0/3030805540 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f6484013470 con 0x7f6488068890 2023-06-23T01:19:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.131+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6488064b50 con 0x7f6488068890 2023-06-23T01:19:40.149 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.134+0000 7f647d7fa700 1 -- 172.21.15.123:0/3030805540 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f6484021060 con 0x7f6488068890 2023-06-23T01:19:40.220 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.216+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e2dc7148-fc1a-48a4-8b06-013164077813", "id": 1} v 0) v1 -- 0x7f64880af760 con 0x7f6488068890 2023-06-23T01:19:40.220 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:40.217+0000 7f8140b431c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:19:40.222 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.218+0000 7f647d7fa700 1 -- 172.21.15.123:0/3030805540 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e2dc7148-fc1a-48a4-8b06-013164077813", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6484017070 con 0x7f6488068890 2023-06-23T01:19:40.222 INFO:teuthology.orchestra.run.smithi123.stdout:1 2023-06-23T01:19:40.223 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.219+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6488068890 msgr2=0x7f64880cc450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.219+0000 7f648f4ad700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6488068890 0x7f64880cc450 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f648400ea50 tx=0x7f64840091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.220+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 shutdown_connections 2023-06-23T01:19:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.220+0000 7f648f4ad700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f64880acff0 0x7f648816d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.220+0000 7f648f4ad700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6488068890 0x7f64880cc450 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.220+0000 7f648f4ad700 1 --2- 172.21.15.123:0/3030805540 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f64880a74c0 0x7f648816cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.220+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 >> 172.21.15.123:0/3030805540 conn(0x7f648806eba0 msgr2=0x7f648806a970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.224 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.220+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 shutdown_connections 2023-06-23T01:19:40.225 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.220+0000 7f648f4ad700 1 -- 172.21.15.123:0/3030805540 wait complete. 2023-06-23T01:19:40.235 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new d30552e4-8c21-4ff9-baf7-07c7ed117bdf 2 2023-06-23T01:19:40.345 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:40.341+0000 7fc2582221c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:19:40.354 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.350+0000 7f49b2215700 1 Processor -- start 2023-06-23T01:19:40.354 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.350+0000 7f49b2215700 1 -- start start 2023-06-23T01:19:40.356 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49b2215700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f49ac151820 0x7f49ac0cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.356 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49b2215700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0c8ef0 con 0x7f49ac151450 2023-06-23T01:19:40.356 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49b2215700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49ac0ccf50 con 0x7f49ac0ccad0 2023-06-23T01:19:40.357 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49b2215700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49ac0c9dc0 con 0x7f49ac151820 2023-06-23T01:19:40.357 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49aaffd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f49ac151820 0x7f49ac0cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.357 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49abfff700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f49ac0ccad0 0x7f49ac0b1210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53800/0 (socket says 172.21.15.123:53800) 2023-06-23T01:19:40.357 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49abfff700 1 -- 172.21.15.123:0/2178792722 learned_addr learned my addr 172.21.15.123:0/2178792722 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-06-23T01:19:40.357 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49aaffd700 1 -- 172.21.15.123:0/2178792722 >> v1:172.21.15.123:6789/0 conn(0x7f49ac0ccad0 legacy=0x7f49ac0b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.357 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49aaffd700 1 -- 172.21.15.123:0/2178792722 >> v1:172.21.15.57:6789/0 conn(0x7f49ac151450 legacy=0x7f49ac0af420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.357 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49aaffd700 1 -- 172.21.15.123:0/2178792722 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49ac0c9140 con 0x7f49ac151820 2023-06-23T01:19:40.358 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49aaffd700 1 --2- 172.21.15.123:0/2178792722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 0x7f49ac0cc590 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f49a00098e0 tx=0x7f49a00091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e4e96a179ce9d1c server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:40.358 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49aa7fc700 1 -- 172.21.15.123:0/2178792722 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0009e80 con 0x7f49ac151820 2023-06-23T01:19:40.358 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49aa7fc700 1 -- 172.21.15.123:0/2178792722 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a000b860 con 0x7f49ac151820 2023-06-23T01:19:40.358 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.351+0000 7f49aa7fc700 1 -- 172.21.15.123:0/2178792722 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000bbe0 con 0x7f49ac151820 2023-06-23T01:19:40.358 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.352+0000 7f49b2215700 1 -- 172.21.15.123:0/2178792722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 msgr2=0x7f49ac0cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.358 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.352+0000 7f49b2215700 1 --2- 172.21.15.123:0/2178792722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 0x7f49ac0cc590 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f49a00098e0 tx=0x7f49a00091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.359 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.352+0000 7f49b2215700 1 -- 172.21.15.123:0/2178792722 shutdown_connections 2023-06-23T01:19:40.359 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.352+0000 7f49b2215700 1 --2- 172.21.15.123:0/2178792722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 0x7f49ac0cc590 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.359 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.352+0000 7f49b2215700 1 -- 172.21.15.123:0/2178792722 >> 172.21.15.123:0/2178792722 conn(0x7f49ac070220 msgr2=0x7f49ac070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.359 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.352+0000 7f49b2215700 1 -- 172.21.15.123:0/2178792722 shutdown_connections 2023-06-23T01:19:40.359 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.352+0000 7f49b2215700 1 -- 172.21.15.123:0/2178792722 wait complete. 2023-06-23T01:19:40.361 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.353+0000 7f49b2215700 1 Processor -- start 2023-06-23T01:19:40.361 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.353+0000 7f49b2215700 1 -- start start 2023-06-23T01:19:40.361 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49b2215700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49ac0ccad0 0x7f49ac0ad9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.362 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49b2215700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49ac151450 0x7f49ac1540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.362 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49b2215700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 0x7f49ac0c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.362 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49b2215700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0c38e0 con 0x7f49ac0ccad0 2023-06-23T01:19:40.362 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49b2215700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac0adf30 con 0x7f49ac151820 2023-06-23T01:19:40.362 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49b2215700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49ac153c20 con 0x7f49ac151450 2023-06-23T01:19:40.362 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49abfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 0x7f49ac0c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49aaffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49ac151450 0x7f49ac1540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49aaffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49ac151450 0x7f49ac1540b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:58538/0 (socket says 172.21.15.123:58538) 2023-06-23T01:19:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49aaffd700 1 -- 172.21.15.123:0/792257458 learned_addr learned my addr 172.21.15.123:0/792257458 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:40.363 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.354+0000 7f49ab7fe700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49ac0ccad0 0x7f49ac0ad9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49aaffd700 1 -- 172.21.15.123:0/792257458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 msgr2=0x7f49ac0c42e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49aaffd700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 0x7f49ac0c42e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49aaffd700 1 -- 172.21.15.123:0/792257458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49ac0ccad0 msgr2=0x7f49ac0ad9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49aaffd700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49ac0ccad0 0x7f49ac0ad9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49aaffd700 1 -- 172.21.15.123:0/792257458 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49a0009420 con 0x7f49ac151450 2023-06-23T01:19:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49abfff700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 0x7f49ac0c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:40.364 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49aaffd700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49ac151450 0x7f49ac1540b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0c8de0 tx=0x7f49a0017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:40.365 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49937fe700 1 -- 172.21.15.123:0/792257458 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a000b9d0 con 0x7f49ac151450 2023-06-23T01:19:40.365 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49937fe700 1 -- 172.21.15.123:0/792257458 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49a000bb30 con 0x7f49ac151450 2023-06-23T01:19:40.365 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49ac1545f0 con 0x7f49ac151450 2023-06-23T01:19:40.365 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49937fe700 1 -- 172.21.15.123:0/792257458 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49a0025650 con 0x7f49ac151450 2023-06-23T01:19:40.366 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49ac0c6650 con 0x7f49ac151450 2023-06-23T01:19:40.366 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49937fe700 1 -- 172.21.15.123:0/792257458 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f49a0023070 con 0x7f49ac151450 2023-06-23T01:19:40.366 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.355+0000 7f49937fe700 1 -- 172.21.15.123:0/792257458 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f49a0011850 con 0x7f49ac151450 2023-06-23T01:19:40.366 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.356+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac153d90 con 0x7f49ac151450 2023-06-23T01:19:40.367 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.358+0000 7f49937fe700 1 -- 172.21.15.123:0/792257458 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f49a002a070 con 0x7f49ac151450 2023-06-23T01:19:40.444 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.441+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d30552e4-8c21-4ff9-baf7-07c7ed117bdf", "id": 2} v 0) v1 -- 0x7f49ac0c6810 con 0x7f49ac151450 2023-06-23T01:19:40.452 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.448+0000 7f49937fe700 1 -- 172.21.15.123:0/792257458 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d30552e4-8c21-4ff9-baf7-07c7ed117bdf", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f49ac0c6810 con 0x7f49ac151450 2023-06-23T01:19:40.453 INFO:teuthology.orchestra.run.smithi123.stdout:2 2023-06-23T01:19:40.454 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49ac151450 msgr2=0x7f49ac1540b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.455 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49ac151450 0x7f49ac1540b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f49ac0c8de0 tx=0x7f49a0017000 comp rx=0 tx=0).stop 2023-06-23T01:19:40.455 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 shutdown_connections 2023-06-23T01:19:40.455 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f49ac151820 0x7f49ac0c42e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.456 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f49ac0ccad0 0x7f49ac0ad9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.456 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 --2- 172.21.15.123:0/792257458 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f49ac151450 0x7f49ac1540b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.456 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 >> 172.21.15.123:0/792257458 conn(0x7f49ac070220 msgr2=0x7f49ac0cc170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.457 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 shutdown_connections 2023-06-23T01:19:40.457 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.449+0000 7f49b2215700 1 -- 172.21.15.123:0/792257458 wait complete. 2023-06-23T01:19:40.464 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new d22728ed-0946-49b6-871d-ab80ec2089c6 3 2023-06-23T01:19:40.488 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:40.485+0000 7fc2582221c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:19:40.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.580+0000 7f9a6b12a700 1 Processor -- start 2023-06-23T01:19:40.584 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.580+0000 7f9a6b12a700 1 -- start start 2023-06-23T01:19:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a6b12a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a230 0x7f9a640cd3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a6b12a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9a64157620 0x7f9a640caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a6b12a700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f9a6415a600 0x7f9a640cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a6b12a700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6415af00 con 0x7f9a6415a600 2023-06-23T01:19:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a6b12a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a6415b1e0 con 0x7f9a64157620 2023-06-23T01:19:40.587 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a6b12a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a6415b4c0 con 0x7f9a6415a230 2023-06-23T01:19:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a68ec6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a230 0x7f9a640cd3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a63fff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9a64157620 0x7f9a640caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a68ec6700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a230 0x7f9a640cd3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:58552/0 (socket says 172.21.15.123:58552) 2023-06-23T01:19:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a68ec6700 1 -- 172.21.15.123:0/2303249653 learned_addr learned my addr 172.21.15.123:0/2303249653 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a63fff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f9a64157620 0x7f9a640caed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49896/0 (socket says 172.21.15.123:49896) 2023-06-23T01:19:40.588 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.581+0000 7f9a696c7700 1 --2- 172.21.15.123:0/2303249653 >> v2:172.21.15.57:3300/0 conn(0x7f9a6415a600 0x7f9a640cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a63fff700 1 -- 172.21.15.123:0/2303249653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a230 msgr2=0x7f9a640cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a63fff700 1 --2- 172.21.15.123:0/2303249653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a230 0x7f9a640cd3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a63fff700 1 -- 172.21.15.123:0/2303249653 >> v2:172.21.15.57:3300/0 conn(0x7f9a6415a600 msgr2=0x7f9a640cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a63fff700 1 --2- 172.21.15.123:0/2303249653 >> v2:172.21.15.57:3300/0 conn(0x7f9a6415a600 0x7f9a640cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a63fff700 1 -- 172.21.15.123:0/2303249653 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a6415b7a0 con 0x7f9a64157620 2023-06-23T01:19:40.589 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a68ec6700 1 --2- 172.21.15.123:0/2303249653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a230 0x7f9a640cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:40.590 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a696c7700 1 --2- 172.21.15.123:0/2303249653 >> v2:172.21.15.57:3300/0 conn(0x7f9a6415a600 0x7f9a640cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:19:40.590 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a63fff700 1 --2- 172.21.15.123:0/2303249653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a64157620 0x7f9a640caed0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9a640cdd80 tx=0x7f9a500091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc45db2eab5d36e6 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:40.590 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a637fe700 1 -- 172.21.15.123:0/2303249653 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50011630 con 0x7f9a64157620 2023-06-23T01:19:40.590 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a637fe700 1 -- 172.21.15.123:0/2303249653 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a50011790 con 0x7f9a64157620 2023-06-23T01:19:40.590 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a637fe700 1 -- 172.21.15.123:0/2303249653 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a50011b30 con 0x7f9a64157620 2023-06-23T01:19:40.590 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 -- 172.21.15.123:0/2303249653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a64157620 msgr2=0x7f9a640caed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.590 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 --2- 172.21.15.123:0/2303249653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a64157620 0x7f9a640caed0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9a640cdd80 tx=0x7f9a500091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.590 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 -- 172.21.15.123:0/2303249653 shutdown_connections 2023-06-23T01:19:40.591 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 --2- 172.21.15.123:0/2303249653 >> v2:172.21.15.57:3300/0 conn(0x7f9a6415a600 0x7f9a640cb410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.591 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 --2- 172.21.15.123:0/2303249653 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a230 0x7f9a640cd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.591 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 --2- 172.21.15.123:0/2303249653 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a64157620 0x7f9a640caed0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.591 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 -- 172.21.15.123:0/2303249653 >> 172.21.15.123:0/2303249653 conn(0x7f9a64067340 msgr2=0x7f9a64067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.591 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 -- 172.21.15.123:0/2303249653 shutdown_connections 2023-06-23T01:19:40.591 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.582+0000 7f9a6b12a700 1 -- 172.21.15.123:0/2303249653 wait complete. 2023-06-23T01:19:40.593 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.583+0000 7f9a6b12a700 1 Processor -- start 2023-06-23T01:19:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.584+0000 7f9a6b12a700 1 -- start start 2023-06-23T01:19:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.584+0000 7f9a6b12a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a64157620 0x7f9a6414fb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.584+0000 7f9a6b12a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a6415a230 0x7f9a640c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.584+0000 7f9a6b12a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a600 0x7f9a640cf570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.584+0000 7f9a6b12a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640c82c0 con 0x7f9a64157620 2023-06-23T01:19:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.584+0000 7f9a6b12a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640c7640 con 0x7f9a6415a230 2023-06-23T01:19:40.594 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.584+0000 7f9a6b12a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a640c7920 con 0x7f9a6415a600 2023-06-23T01:19:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.584+0000 7f9a696c7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a600 0x7f9a640cf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a63fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a6415a230 0x7f9a640c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a696c7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a600 0x7f9a640cf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:58566/0 (socket says 172.21.15.123:58566) 2023-06-23T01:19:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a696c7700 1 -- 172.21.15.123:0/4230492788 learned_addr learned my addr 172.21.15.123:0/4230492788 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a68ec6700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a64157620 0x7f9a6414fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a696c7700 1 -- 172.21.15.123:0/4230492788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a6415a230 msgr2=0x7f9a640c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a696c7700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a6415a230 0x7f9a640c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.595 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a696c7700 1 -- 172.21.15.123:0/4230492788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a64157620 msgr2=0x7f9a6414fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a696c7700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a64157620 0x7f9a6414fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a696c7700 1 -- 172.21.15.123:0/4230492788 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a58009480 con 0x7f9a6415a600 2023-06-23T01:19:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a63fff700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a6415a230 0x7f9a640c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a696c7700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a600 0x7f9a640cf570 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009fc0 tx=0x7f9a5c0099b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a60ff9700 1 -- 172.21.15.123:0/4230492788 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c00dbe0 con 0x7f9a6415a600 2023-06-23T01:19:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a60ff9700 1 -- 172.21.15.123:0/4230492788 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a5c00f040 con 0x7f9a6415a600 2023-06-23T01:19:40.596 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a60ff9700 1 -- 172.21.15.123:0/4230492788 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a5c015440 con 0x7f9a6415a600 2023-06-23T01:19:40.597 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a50009420 con 0x7f9a6415a600 2023-06-23T01:19:40.597 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a640c7ee0 con 0x7f9a6415a600 2023-06-23T01:19:40.597 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a60ff9700 1 -- 172.21.15.123:0/4230492788 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9a5c015870 con 0x7f9a6415a600 2023-06-23T01:19:40.597 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.585+0000 7f9a60ff9700 1 -- 172.21.15.123:0/4230492788 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f9a5c01e850 con 0x7f9a6415a600 2023-06-23T01:19:40.597 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.586+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a64064b50 con 0x7f9a6415a600 2023-06-23T01:19:40.597 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.588+0000 7f9a60ff9700 1 -- 172.21.15.123:0/4230492788 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f9a5c01a070 con 0x7f9a6415a600 2023-06-23T01:19:40.673 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.670+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d22728ed-0946-49b6-871d-ab80ec2089c6", "id": 3} v 0) v1 -- 0x7f9a6415ba80 con 0x7f9a6415a600 2023-06-23T01:19:40.686 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.682+0000 7f9a60ff9700 1 -- 172.21.15.123:0/4230492788 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d22728ed-0946-49b6-871d-ab80ec2089c6", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9a5c015d50 con 0x7f9a6415a600 2023-06-23T01:19:40.686 INFO:teuthology.orchestra.run.smithi123.stdout:3 2023-06-23T01:19:40.687 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.683+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a600 msgr2=0x7f9a640cf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.687 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.683+0000 7f9a6b12a700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a600 0x7f9a640cf570 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f9a5c009fc0 tx=0x7f9a5c0099b0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.688 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.683+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 shutdown_connections 2023-06-23T01:19:40.688 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.683+0000 7f9a6b12a700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f9a6415a600 0x7f9a640cf570 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.688 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.683+0000 7f9a6b12a700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f9a64157620 0x7f9a6414fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.688 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.683+0000 7f9a6b12a700 1 --2- 172.21.15.123:0/4230492788 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9a6415a230 0x7f9a640c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.688 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.683+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 >> 172.21.15.123:0/4230492788 conn(0x7f9a64067340 msgr2=0x7f9a640cc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.688 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.683+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 shutdown_connections 2023-06-23T01:19:40.688 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.684+0000 7f9a6b12a700 1 -- 172.21.15.123:0/4230492788 wait complete. 2023-06-23T01:19:40.699 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new a39e2e51-40b1-4c91-b045-f6f030dd6546 4 2023-06-23T01:19:40.744 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:40.740+0000 7f8140b431c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:19:40.818 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.814+0000 7f35f593a700 1 Processor -- start 2023-06-23T01:19:40.818 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.814+0000 7f35f593a700 1 -- start start 2023-06-23T01:19:40.826 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35f593a700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f35f0155ad0 0x7f35f006e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.827 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35f593a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f015a020 0x7f35f00cc490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.827 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35f593a700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f006e4a0 con 0x7f35f0155700 2023-06-23T01:19:40.827 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35f593a700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35f00c7cf0 con 0x7f35f0155ad0 2023-06-23T01:19:40.827 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35f593a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f006d890 con 0x7f35f015a020 2023-06-23T01:19:40.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35ef7fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f015a020 0x7f35f00cc490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35ef7fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f015a020 0x7f35f00cc490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:58568/0 (socket says 172.21.15.123:58568) 2023-06-23T01:19:40.828 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35ef7fe700 1 -- 172.21.15.123:0/1771343319 learned_addr learned my addr 172.21.15.123:0/1771343319 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:40.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35ee7fc700 1 --2- 172.21.15.123:0/1771343319 >> v2:172.21.15.57:3300/0 conn(0x7f35f0155ad0 0x7f35f006e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35ef7fe700 1 -- 172.21.15.123:0/1771343319 >> v2:172.21.15.57:3300/0 conn(0x7f35f0155ad0 msgr2=0x7f35f006e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:40.829 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35ef7fe700 1 --2- 172.21.15.123:0/1771343319 >> v2:172.21.15.57:3300/0 conn(0x7f35f0155ad0 0x7f35f006e930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.830 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35ef7fe700 1 -- 172.21.15.123:0/1771343319 >> v1:172.21.15.57:6789/0 conn(0x7f35f0155700 legacy=0x7f35f00c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.830 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.815+0000 7f35ef7fe700 1 -- 172.21.15.123:0/1771343319 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f006db70 con 0x7f35f015a020 2023-06-23T01:19:40.830 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35ef7fe700 1 --2- 172.21.15.123:0/1771343319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f015a020 0x7f35f00cc490 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009f20 tx=0x7f35e8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=215fa64697bcc969 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:40.830 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35edffb700 1 -- 172.21.15.123:0/1771343319 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e800e070 con 0x7f35f015a020 2023-06-23T01:19:40.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35edffb700 1 -- 172.21.15.123:0/1771343319 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e800b9d0 con 0x7f35f015a020 2023-06-23T01:19:40.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35edffb700 1 -- 172.21.15.123:0/1771343319 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e80092e0 con 0x7f35f015a020 2023-06-23T01:19:40.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35f593a700 1 -- 172.21.15.123:0/1771343319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f015a020 msgr2=0x7f35f00cc490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35f593a700 1 --2- 172.21.15.123:0/1771343319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f015a020 0x7f35f00cc490 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f35e8009f20 tx=0x7f35e8009840 comp rx=0 tx=0).stop 2023-06-23T01:19:40.831 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35f593a700 1 -- 172.21.15.123:0/1771343319 shutdown_connections 2023-06-23T01:19:40.832 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35f593a700 1 --2- 172.21.15.123:0/1771343319 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f015a020 0x7f35f00cc490 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.832 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35f593a700 1 --2- 172.21.15.123:0/1771343319 >> v2:172.21.15.57:3300/0 conn(0x7f35f0155ad0 0x7f35f006e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.832 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35f593a700 1 -- 172.21.15.123:0/1771343319 >> 172.21.15.123:0/1771343319 conn(0x7f35f0070220 msgr2=0x7f35f0070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.832 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35f593a700 1 -- 172.21.15.123:0/1771343319 shutdown_connections 2023-06-23T01:19:40.832 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.816+0000 7f35f593a700 1 -- 172.21.15.123:0/1771343319 wait complete. 2023-06-23T01:19:40.832 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35f593a700 1 Processor -- start 2023-06-23T01:19:40.833 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35f593a700 1 -- start start 2023-06-23T01:19:40.833 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35f593a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f0155700 0x7f35f0150550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.833 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35f593a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0155ad0 0x7f35f00c7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.833 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35f593a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f015a020 0x7f35f01580b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:40.833 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35f593a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f01517b0 con 0x7f35f015a020 2023-06-23T01:19:40.833 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35f593a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0150c30 con 0x7f35f0155ad0 2023-06-23T01:19:40.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35f593a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f0150f10 con 0x7f35f0155700 2023-06-23T01:19:40.835 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35ee7fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0155ad0 0x7f35f00c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35eeffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f0155700 0x7f35f0150550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35ee7fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0155ad0 0x7f35f00c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49908/0 (socket says 172.21.15.123:49908) 2023-06-23T01:19:40.836 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35ee7fc700 1 -- 172.21.15.123:0/1583517056 learned_addr learned my addr 172.21.15.123:0/1583517056 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:40.837 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35eeffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f0155700 0x7f35f0150550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:58574/0 (socket says 172.21.15.123:58574) 2023-06-23T01:19:40.837 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.818+0000 7f35ef7fe700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f015a020 0x7f35f01580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:40.837 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35ee7fc700 1 -- 172.21.15.123:0/1583517056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f0155700 msgr2=0x7f35f0150550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.838 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35ee7fc700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f0155700 0x7f35f0150550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.838 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35ee7fc700 1 -- 172.21.15.123:0/1583517056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f015a020 msgr2=0x7f35f01580b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.838 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35ee7fc700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f015a020 0x7f35f01580b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.839 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35ee7fc700 1 -- 172.21.15.123:0/1583517056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8009ae0 con 0x7f35f0155ad0 2023-06-23T01:19:40.839 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35eeffd700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f0155700 0x7f35f0150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:40.839 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35ef7fe700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f015a020 0x7f35f01580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:40.839 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35ee7fc700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0155ad0 0x7f35f00c7040 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f35e400ec80 tx=0x7f35e40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:40.840 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35d6ffd700 1 -- 172.21.15.123:0/1583517056 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e400c850 con 0x7f35f0155ad0 2023-06-23T01:19:40.840 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35d6ffd700 1 -- 172.21.15.123:0/1583517056 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e400c9b0 con 0x7f35f0155ad0 2023-06-23T01:19:40.840 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35d6ffd700 1 -- 172.21.15.123:0/1583517056 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e400cd50 con 0x7f35f0155ad0 2023-06-23T01:19:40.840 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f01511f0 con 0x7f35f0155ad0 2023-06-23T01:19:40.841 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.819+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f00c7580 con 0x7f35f0155ad0 2023-06-23T01:19:40.841 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.820+0000 7f35d6ffd700 1 -- 172.21.15.123:0/1583517056 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f35e4013900 con 0x7f35f0155ad0 2023-06-23T01:19:40.841 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.820+0000 7f35d6ffd700 1 -- 172.21.15.123:0/1583517056 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f35e401caa0 con 0x7f35f0155ad0 2023-06-23T01:19:40.841 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.820+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f0064b50 con 0x7f35f0155ad0 2023-06-23T01:19:40.842 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.822+0000 7f35d6ffd700 1 -- 172.21.15.123:0/1583517056 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f35e4021060 con 0x7f35f0155ad0 2023-06-23T01:19:40.842 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:40.836+0000 7fc2582221c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:19:40.887 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:40.883+0000 7f8140b431c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:19:40.909 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.905+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "a39e2e51-40b1-4c91-b045-f6f030dd6546", "id": 4} v 0) v1 -- 0x7f35f01681c0 con 0x7f35f0155ad0 2023-06-23T01:19:40.921 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.917+0000 7f35d6ffd700 1 -- 172.21.15.123:0/1583517056 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a39e2e51-40b1-4c91-b045-f6f030dd6546", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f35e4017070 con 0x7f35f0155ad0 2023-06-23T01:19:40.921 INFO:teuthology.orchestra.run.smithi123.stdout:4 2023-06-23T01:19:40.923 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0155ad0 msgr2=0x7f35f00c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:40.923 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0155ad0 0x7f35f00c7040 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f35e400ec80 tx=0x7f35e40091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.923 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 shutdown_connections 2023-06-23T01:19:40.924 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f35f015a020 0x7f35f01580b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.924 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f35f0155700 0x7f35f0150550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.924 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 --2- 172.21.15.123:0/1583517056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f35f0155ad0 0x7f35f00c7040 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:40.924 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 >> 172.21.15.123:0/1583517056 conn(0x7f35f0070220 msgr2=0x7f35f00639c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:40.925 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 shutdown_connections 2023-06-23T01:19:40.925 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:40.918+0000 7f35f593a700 1 -- 172.21.15.123:0/1583517056 wait complete. 2023-06-23T01:19:40.935 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 855f67e8-1b47-4fb7-a8c2-2f5df954a8ab 5 2023-06-23T01:19:41.052 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.049+0000 7fed6ac72700 1 Processor -- start 2023-06-23T01:19:41.053 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.049+0000 7fed6ac72700 1 -- start start 2023-06-23T01:19:41.055 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.049+0000 7fed6ac72700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fed640ccba0 0x7fed6406e930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.055 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.049+0000 7fed6ac72700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed6406e4a0 con 0x7fed64155ad0 2023-06-23T01:19:41.055 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.049+0000 7fed6ac72700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed640c7cf0 con 0x7fed64155700 2023-06-23T01:19:41.055 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.049+0000 7fed6ac72700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed6406d890 con 0x7fed640ccba0 2023-06-23T01:19:41.055 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed68a0e700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fed64155700 0x7fed640c75e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53808/0 (socket says 172.21.15.123:53808) 2023-06-23T01:19:41.055 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed6920f700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fed640ccba0 0x7fed6406e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.056 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed68a0e700 1 -- 172.21.15.123:0/67165883 learned_addr learned my addr 172.21.15.123:0/67165883 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-06-23T01:19:41.056 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed6920f700 1 -- 172.21.15.123:0/67165883 >> v1:172.21.15.123:6789/0 conn(0x7fed64155700 legacy=0x7fed640c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.056 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed6920f700 1 -- 172.21.15.123:0/67165883 >> v1:172.21.15.57:6789/0 conn(0x7fed64155ad0 legacy=0x7fed640cc490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.056 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed6920f700 1 -- 172.21.15.123:0/67165883 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6406db70 con 0x7fed640ccba0 2023-06-23T01:19:41.056 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed6920f700 1 --2- 172.21.15.123:0/67165883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed640ccba0 0x7fed6406e930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00af70 tx=0x7fed5c00a890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b616fc26e5afa03e server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:41.056 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed637fe700 1 -- 172.21.15.123:0/67165883 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c00f070 con 0x7fed640ccba0 2023-06-23T01:19:41.056 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed637fe700 1 -- 172.21.15.123:0/67165883 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c00a2e0 con 0x7fed640ccba0 2023-06-23T01:19:41.057 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.050+0000 7fed637fe700 1 -- 172.21.15.123:0/67165883 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c00cd20 con 0x7fed640ccba0 2023-06-23T01:19:41.057 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.051+0000 7fed6ac72700 1 -- 172.21.15.123:0/67165883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed640ccba0 msgr2=0x7fed6406e930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.057 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.051+0000 7fed6ac72700 1 --2- 172.21.15.123:0/67165883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed640ccba0 0x7fed6406e930 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00af70 tx=0x7fed5c00a890 comp rx=0 tx=0).stop 2023-06-23T01:19:41.057 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.051+0000 7fed6ac72700 1 -- 172.21.15.123:0/67165883 shutdown_connections 2023-06-23T01:19:41.057 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.051+0000 7fed6ac72700 1 --2- 172.21.15.123:0/67165883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed640ccba0 0x7fed6406e930 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.057 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.051+0000 7fed6ac72700 1 -- 172.21.15.123:0/67165883 >> 172.21.15.123:0/67165883 conn(0x7fed64070220 msgr2=0x7fed64070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:41.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.051+0000 7fed6ac72700 1 -- 172.21.15.123:0/67165883 shutdown_connections 2023-06-23T01:19:41.058 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.051+0000 7fed6ac72700 1 -- 172.21.15.123:0/67165883 wait complete. 2023-06-23T01:19:41.060 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.052+0000 7fed6ac72700 1 Processor -- start 2023-06-23T01:19:41.061 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.052+0000 7fed6ac72700 1 -- start start 2023-06-23T01:19:41.061 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6ac72700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed640ccba0 0x7fed641692a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.061 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6ac72700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed64155700 0x7fed6415acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.061 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6ac72700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed64155ad0 0x7fed640bfe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.061 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6ac72700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed641521b0 con 0x7fed64155700 2023-06-23T01:19:41.061 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6ac72700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed64152770 con 0x7fed64155ad0 2023-06-23T01:19:41.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6ac72700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed6415b7a0 con 0x7fed640ccba0 2023-06-23T01:19:41.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed68a0e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed640ccba0 0x7fed641692a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6920f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed64155ad0 0x7fed640bfe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6920f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed64155ad0 0x7fed640bfe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49918/0 (socket says 172.21.15.123:49918) 2023-06-23T01:19:41.062 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed68a0e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed640ccba0 0x7fed641692a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:58576/0 (socket says 172.21.15.123:58576) 2023-06-23T01:19:41.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6920f700 1 -- 172.21.15.123:0/17791676 learned_addr learned my addr 172.21.15.123:0/17791676 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:41.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed63fff700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed64155700 0x7fed6415acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6920f700 1 -- 172.21.15.123:0/17791676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed640ccba0 msgr2=0x7fed641692a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.063 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6920f700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed640ccba0 0x7fed641692a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6920f700 1 -- 172.21.15.123:0/17791676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed64155700 msgr2=0x7fed6415acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6920f700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed64155700 0x7fed6415acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed6920f700 1 -- 172.21.15.123:0/17791676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5c00ab30 con 0x7fed64155ad0 2023-06-23T01:19:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.053+0000 7fed68a0e700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed640ccba0 0x7fed641692a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:41.064 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.054+0000 7fed6920f700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed64155ad0 0x7fed640bfe10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00ab00 tx=0x7fed5c019000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:41.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.054+0000 7fed60ff9700 1 -- 172.21.15.123:0/17791676 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c009d70 con 0x7fed64155ad0 2023-06-23T01:19:41.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.054+0000 7fed60ff9700 1 -- 172.21.15.123:0/17791676 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed5c027410 con 0x7fed64155ad0 2023-06-23T01:19:41.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.054+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed6415ba80 con 0x7fed64155ad0 2023-06-23T01:19:41.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.054+0000 7fed60ff9700 1 -- 172.21.15.123:0/17791676 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed5c0277d0 con 0x7fed64155ad0 2023-06-23T01:19:41.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.054+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed6414d130 con 0x7fed64155ad0 2023-06-23T01:19:41.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.054+0000 7fed60ff9700 1 -- 172.21.15.123:0/17791676 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fed5c003020 con 0x7fed64155ad0 2023-06-23T01:19:41.065 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.054+0000 7fed60ff9700 1 -- 172.21.15.123:0/17791676 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fed5c027e10 con 0x7fed64155ad0 2023-06-23T01:19:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.055+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed64064b50 con 0x7fed64155ad0 2023-06-23T01:19:41.066 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.058+0000 7fed60ff9700 1 -- 172.21.15.123:0/17791676 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fed5c0466e0 con 0x7fed64155ad0 2023-06-23T01:19:41.146 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.141+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "855f67e8-1b47-4fb7-a8c2-2f5df954a8ab", "id": 5} v 0) v1 -- 0x7fed641528b0 con 0x7fed64155ad0 2023-06-23T01:19:41.153 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.149+0000 7fed60ff9700 1 -- 172.21.15.123:0/17791676 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "855f67e8-1b47-4fb7-a8c2-2f5df954a8ab", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fed5c035030 con 0x7fed64155ad0 2023-06-23T01:19:41.153 INFO:teuthology.orchestra.run.smithi123.stdout:5 2023-06-23T01:19:41.154 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.150+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed64155ad0 msgr2=0x7fed640bfe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.154 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.150+0000 7fed6ac72700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed64155ad0 0x7fed640bfe10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fed5c00ab00 tx=0x7fed5c019000 comp rx=0 tx=0).stop 2023-06-23T01:19:41.154 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.150+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 shutdown_connections 2023-06-23T01:19:41.154 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.150+0000 7fed6ac72700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed64155ad0 0x7fed640bfe10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.155 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.150+0000 7fed6ac72700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fed640ccba0 0x7fed641692a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.155 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.150+0000 7fed6ac72700 1 --2- 172.21.15.123:0/17791676 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fed64155700 0x7fed6415acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.155 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.150+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 >> 172.21.15.123:0/17791676 conn(0x7fed64070220 msgr2=0x7fed640af690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:41.155 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.151+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 shutdown_connections 2023-06-23T01:19:41.155 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.151+0000 7fed6ac72700 1 -- 172.21.15.123:0/17791676 wait complete. 2023-06-23T01:19:41.166 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new a74edd8e-c135-473e-9394-f214d0405572 6 2023-06-23T01:19:41.243 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:41.240+0000 7f8140b431c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:19:41.288 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.284+0000 7f2679fc0700 1 Processor -- start 2023-06-23T01:19:41.288 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.284+0000 7f2679fc0700 1 -- start start 2023-06-23T01:19:41.290 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f2679fc0700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2674151450 0x7f2674151820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.290 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f2679fc0700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26740c8ef0 con 0x7f26740af8a0 2023-06-23T01:19:41.290 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f2679fc0700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26740c9140 con 0x7f26740af420 2023-06-23T01:19:41.290 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f2679fc0700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26740c9dc0 con 0x7f2674151450 2023-06-23T01:19:41.290 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f26737fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2674151450 0x7f2674151820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.290 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f26737fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f2674151450 0x7f2674151820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49932/0 (socket says 172.21.15.123:49932) 2023-06-23T01:19:41.291 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f2672ffd700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f26740af420 0x7f26740cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53820/0 (socket says 172.21.15.123:53820) 2023-06-23T01:19:41.291 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f26737fe700 1 -- 172.21.15.123:0/3688086044 learned_addr learned my addr 172.21.15.123:0/3688086044 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:41.291 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.285+0000 7f26737fe700 1 -- 172.21.15.123:0/3688086044 >> v1:172.21.15.123:6789/0 conn(0x7f26740af420 legacy=0x7f26740cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.291 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.286+0000 7f26737fe700 1 -- 172.21.15.123:0/3688086044 >> v1:172.21.15.57:6789/0 conn(0x7f26740af8a0 legacy=0x7f26740b1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.291 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.286+0000 7f26737fe700 1 -- 172.21.15.123:0/3688086044 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26740c9420 con 0x7f2674151450 2023-06-23T01:19:41.291 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.286+0000 7f26737fe700 1 --2- 172.21.15.123:0/3688086044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 0x7f2674151820 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2664009f20 tx=0x7f2664009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9819db3bb2ab7ae8 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:41.291 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.286+0000 7f26727fc700 1 -- 172.21.15.123:0/3688086044 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f266400e070 con 0x7f2674151450 2023-06-23T01:19:41.292 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.286+0000 7f26727fc700 1 -- 172.21.15.123:0/3688086044 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f266400ba80 con 0x7f2674151450 2023-06-23T01:19:41.292 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.286+0000 7f26727fc700 1 -- 172.21.15.123:0/3688086044 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26640092e0 con 0x7f2674151450 2023-06-23T01:19:41.294 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.286+0000 7f2679fc0700 1 -- 172.21.15.123:0/3688086044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 msgr2=0x7f2674151820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.294 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.286+0000 7f2679fc0700 1 --2- 172.21.15.123:0/3688086044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 0x7f2674151820 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f2664009f20 tx=0x7f2664009840 comp rx=0 tx=0).stop 2023-06-23T01:19:41.294 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.287+0000 7f2679fc0700 1 -- 172.21.15.123:0/3688086044 shutdown_connections 2023-06-23T01:19:41.294 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.287+0000 7f2679fc0700 1 --2- 172.21.15.123:0/3688086044 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 0x7f2674151820 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.294 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.287+0000 7f2679fc0700 1 -- 172.21.15.123:0/3688086044 >> 172.21.15.123:0/3688086044 conn(0x7f2674070220 msgr2=0x7f2674070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:41.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.287+0000 7f2679fc0700 1 -- 172.21.15.123:0/3688086044 shutdown_connections 2023-06-23T01:19:41.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.287+0000 7f2679fc0700 1 -- 172.21.15.123:0/3688086044 wait complete. 2023-06-23T01:19:41.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.288+0000 7f2679fc0700 1 Processor -- start 2023-06-23T01:19:41.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.288+0000 7f2679fc0700 1 -- start start 2023-06-23T01:19:41.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2679fc0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26740af420 0x7f26740b2130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2679fc0700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26740af8a0 0x7f267414ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.295 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2679fc0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 0x7f26740c21f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2679fc0700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26740c59a0 con 0x7f26740af8a0 2023-06-23T01:19:41.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2679fc0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f267406eba0 con 0x7f2674151450 2023-06-23T01:19:41.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2679fc0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f267406db00 con 0x7f26740af420 2023-06-23T01:19:41.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26740af420 0x7f26740b2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.296 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26740af420 0x7f26740b2130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.123:58582/0 (socket says 172.21.15.123:58582) 2023-06-23T01:19:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2673fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 0x7f26740c21f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 -- 172.21.15.123:0/3301120898 learned_addr learned my addr 172.21.15.123:0/3301120898 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2672ffd700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26740af8a0 0x7f267414ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 -- 172.21.15.123:0/3301120898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 msgr2=0x7f26740c21f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 0x7f26740c21f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 -- 172.21.15.123:0/3301120898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26740af8a0 msgr2=0x7f267414ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26740af8a0 0x7f267414ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.297 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 -- 172.21.15.123:0/3301120898 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2664009ae0 con 0x7f26740af420 2023-06-23T01:19:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f2673fff700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 0x7f26740c21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f26737fe700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26740af420 0x7f26740b2130 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f266400b6c0 tx=0x7f2664018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f266b7fe700 1 -- 172.21.15.123:0/3301120898 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26640089b0 con 0x7f26740af420 2023-06-23T01:19:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.289+0000 7f266b7fe700 1 -- 172.21.15.123:0/3301120898 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2664008b10 con 0x7f26740af420 2023-06-23T01:19:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.290+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f267406dde0 con 0x7f26740af420 2023-06-23T01:19:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.290+0000 7f266b7fe700 1 -- 172.21.15.123:0/3301120898 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2664025460 con 0x7f26740af420 2023-06-23T01:19:41.298 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.290+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f267406e3a0 con 0x7f26740af420 2023-06-23T01:19:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.291+0000 7f266b7fe700 1 -- 172.21.15.123:0/3301120898 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f2664003040 con 0x7f26740af420 2023-06-23T01:19:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.291+0000 7f266b7fe700 1 -- 172.21.15.123:0/3301120898 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f2664025bb0 con 0x7f26740af420 2023-06-23T01:19:41.299 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.291+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2674064b50 con 0x7f26740af420 2023-06-23T01:19:41.300 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.293+0000 7f266b7fe700 1 -- 172.21.15.123:0/3301120898 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f266402a020 con 0x7f26740af420 2023-06-23T01:19:41.379 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.376+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a74edd8e-c135-473e-9394-f214d0405572", "id": 6} v 0) v1 -- 0x7f26740c9790 con 0x7f26740af420 2023-06-23T01:19:41.387 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.383+0000 7f266b7fe700 1 -- 172.21.15.123:0/3301120898 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a74edd8e-c135-473e-9394-f214d0405572", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2664020070 con 0x7f26740af420 2023-06-23T01:19:41.387 INFO:teuthology.orchestra.run.smithi123.stdout:6 2023-06-23T01:19:41.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26740af420 msgr2=0x7f26740b2130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26740af420 0x7f26740b2130 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f266400b6c0 tx=0x7f2664018000 comp rx=0 tx=0).stop 2023-06-23T01:19:41.388 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 shutdown_connections 2023-06-23T01:19:41.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2674151450 0x7f26740c21f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f26740af420 0x7f26740b2130 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 --2- 172.21.15.123:0/3301120898 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f26740af8a0 0x7f267414ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 >> 172.21.15.123:0/3301120898 conn(0x7f2674070220 msgr2=0x7f267415a320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:41.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 shutdown_connections 2023-06-23T01:19:41.389 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.384+0000 7f2679fc0700 1 -- 172.21.15.123:0/3301120898 wait complete. 2023-06-23T01:19:41.399 DEBUG:teuthology.orchestra.run.smithi123:> sudo ceph --cluster ceph osd new 26725e66-b047-4e78-afc6-352a8ea70559 7 2023-06-23T01:19:41.518 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.514+0000 7fcdaea62700 1 Processor -- start 2023-06-23T01:19:41.518 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.514+0000 7fcdaea62700 1 -- start start 2023-06-23T01:19:41.521 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcdaea62700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcda80c7510 0x7fcda80afc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.521 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcdaea62700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fcda8140b60 0x7fcda8068350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.521 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcdaea62700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcda80cabd0 con 0x7fcda8140790 2023-06-23T01:19:41.521 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcdaea62700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda80c91a0 con 0x7fcda8140b60 2023-06-23T01:19:41.522 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcdaea62700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcda80c9720 con 0x7fcda80c7510 2023-06-23T01:19:41.522 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda77fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcda80c7510 0x7fcda80afc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.522 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda7fff700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7fcda8140790 0x7fcda815b5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.123:53824/0 (socket says 172.21.15.123:53824) 2023-06-23T01:19:41.522 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda77fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fcda80c7510 0x7fcda80afc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49948/0 (socket says 172.21.15.123:49948) 2023-06-23T01:19:41.523 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda7fff700 1 -- 172.21.15.123:0/2849909842 learned_addr learned my addr 172.21.15.123:0/2849909842 (peer_addr_for_me v1:172.21.15.123:0/0) 2023-06-23T01:19:41.523 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcdacfff700 1 --2- 172.21.15.123:0/2849909842 >> v2:172.21.15.57:3300/0 conn(0x7fcda8140b60 0x7fcda8068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.523 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda77fe700 1 -- 172.21.15.123:0/2849909842 >> v2:172.21.15.57:3300/0 conn(0x7fcda8140b60 msgr2=0x7fcda8068350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda77fe700 1 --2- 172.21.15.123:0/2849909842 >> v2:172.21.15.57:3300/0 conn(0x7fcda8140b60 0x7fcda8068350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda77fe700 1 -- 172.21.15.123:0/2849909842 >> v1:172.21.15.123:6789/0 conn(0x7fcda8140790 legacy=0x7fcda815b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda77fe700 1 -- 172.21.15.123:0/2849909842 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcda80c9440 con 0x7fcda80c7510 2023-06-23T01:19:41.524 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda6ffd700 1 -- 172.21.15.123:0/2849909842 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3549430690 0 0) 0x7fcda80cabd0 con 0x7fcda8140790 2023-06-23T01:19:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda77fe700 1 --2- 172.21.15.123:0/2849909842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80afc90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0098e0 tx=0x7fcd9c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96e2e2bfb7a031d1 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda6ffd700 1 -- 172.21.15.123:0/2849909842 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c011630 con 0x7fcda80c7510 2023-06-23T01:19:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda6ffd700 1 -- 172.21.15.123:0/2849909842 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd9c011790 con 0x7fcda80c7510 2023-06-23T01:19:41.525 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.515+0000 7fcda6ffd700 1 -- 172.21.15.123:0/2849909842 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9c011b30 con 0x7fcda80c7510 2023-06-23T01:19:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.516+0000 7fcdaea62700 1 -- 172.21.15.123:0/2849909842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 msgr2=0x7fcda80afc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.526 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.516+0000 7fcdaea62700 1 --2- 172.21.15.123:0/2849909842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80afc90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fcd9c0098e0 tx=0x7fcd9c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.527 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.516+0000 7fcdaea62700 1 -- 172.21.15.123:0/2849909842 shutdown_connections 2023-06-23T01:19:41.527 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.516+0000 7fcdaea62700 1 --2- 172.21.15.123:0/2849909842 >> v2:172.21.15.57:3300/0 conn(0x7fcda8140b60 0x7fcda8068350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.516+0000 7fcdaea62700 1 --2- 172.21.15.123:0/2849909842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80afc90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.516+0000 7fcdaea62700 1 -- 172.21.15.123:0/2849909842 >> 172.21.15.123:0/2849909842 conn(0x7fcda80ca410 msgr2=0x7fcda806e110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:41.528 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.516+0000 7fcdaea62700 1 -- 172.21.15.123:0/2849909842 shutdown_connections 2023-06-23T01:19:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.516+0000 7fcdaea62700 1 -- 172.21.15.123:0/2849909842 wait complete. 2023-06-23T01:19:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.517+0000 7fcdaea62700 1 Processor -- start 2023-06-23T01:19:41.529 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.517+0000 7fcdaea62700 1 -- start start 2023-06-23T01:19:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcdaea62700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80a9880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcdaea62700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcda8140790 0x7fcda81569c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.533 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcdaea62700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcda8140b60 0x7fcda813eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcdaea62700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda80ac2e0 con 0x7fcda8140790 2023-06-23T01:19:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcdaea62700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcda80a9dc0 con 0x7fcda80c7510 2023-06-23T01:19:41.534 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcdaea62700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcda80a9ff0 con 0x7fcda8140b60 2023-06-23T01:19:41.535 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcda7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.535 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcdacfff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcda8140b60 0x7fcda813eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.535 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcda7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80a9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:49962/0 (socket says 172.21.15.123:49962) 2023-06-23T01:19:41.536 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcda7fff700 1 -- 172.21.15.123:0/2414609726 learned_addr learned my addr 172.21.15.123:0/2414609726 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-06-23T01:19:41.536 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.518+0000 7fcda77fe700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcda8140790 0x7fcda81569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.536 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcda7fff700 1 -- 172.21.15.123:0/2414609726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcda8140b60 msgr2=0x7fcda813eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.537 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcda7fff700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcda8140b60 0x7fcda813eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.537 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcda7fff700 1 -- 172.21.15.123:0/2414609726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcda8140790 msgr2=0x7fcda81569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.537 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcda7fff700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcda8140790 0x7fcda81569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.537 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcda7fff700 1 -- 172.21.15.123:0/2414609726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd9c009420 con 0x7fcda80c7510 2023-06-23T01:19:41.538 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcdacfff700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcda8140b60 0x7fcda813eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:41.538 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcda7fff700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80a9880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800c7e0 tx=0x7fcd98007df0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:41.538 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcd8ffff700 1 -- 172.21.15.123:0/2414609726 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98005b30 con 0x7fcda80c7510 2023-06-23T01:19:41.538 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcd8ffff700 1 -- 172.21.15.123:0/2414609726 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd98005c90 con 0x7fcda80c7510 2023-06-23T01:19:41.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcda806a520 con 0x7fcda80c7510 2023-06-23T01:19:41.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcd8ffff700 1 -- 172.21.15.123:0/2414609726 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98007300 con 0x7fcda80c7510 2023-06-23T01:19:41.539 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcda8069bf0 con 0x7fcda80c7510 2023-06-23T01:19:41.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcd8ffff700 1 -- 172.21.15.123:0/2414609726 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcd9801c440 con 0x7fcda80c7510 2023-06-23T01:19:41.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.519+0000 7fcd8ffff700 1 -- 172.21.15.123:0/2414609726 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7fcd98005e00 con 0x7fcda80c7510 2023-06-23T01:19:41.540 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.520+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcda8064b50 con 0x7fcda80c7510 2023-06-23T01:19:41.541 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.522+0000 7fcd8ffff700 1 -- 172.21.15.123:0/2414609726 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fcd98017070 con 0x7fcda80c7510 2023-06-23T01:19:41.595 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:41.591+0000 7fc2582221c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:19:41.607 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.604+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd new", "uuid": "26725e66-b047-4e78-afc6-352a8ea70559", "id": 7} v 0) v1 -- 0x7fcda80c7a80 con 0x7fcda80c7510 2023-06-23T01:19:41.619 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.615+0000 7fcd8ffff700 1 -- 172.21.15.123:0/2414609726 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "26725e66-b047-4e78-afc6-352a8ea70559", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcd9801c830 con 0x7fcda80c7510 2023-06-23T01:19:41.619 INFO:teuthology.orchestra.run.smithi123.stdout:7 2023-06-23T01:19:41.620 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 msgr2=0x7fcda80a9880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.621 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80a9880 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800c7e0 tx=0x7fcd98007df0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.621 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 shutdown_connections 2023-06-23T01:19:41.621 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fcda8140b60 0x7fcda813eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.621 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcda80c7510 0x7fcda80a9880 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.621 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 --2- 172.21.15.123:0/2414609726 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fcda8140790 0x7fcda81569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.621 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 >> 172.21.15.123:0/2414609726 conn(0x7fcda80ca410 msgr2=0x7fcda80c2890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:41.621 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 shutdown_connections 2023-06-23T01:19:41.622 INFO:teuthology.orchestra.run.smithi123.stderr:2023-06-23T01:19:41.616+0000 7fcdaea62700 1 -- 172.21.15.123:0/2414609726 wait complete. 2023-06-23T01:19:41.631 INFO:tasks.ceph.osd.0:Restarting daemon 2023-06-23T01:19:41.632 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-06-23T01:19:41.635 INFO:tasks.ceph.osd.0:Started 2023-06-23T01:19:41.636 INFO:tasks.ceph.osd.1:Restarting daemon 2023-06-23T01:19:41.637 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-06-23T01:19:41.639 INFO:tasks.ceph.osd.1:Started 2023-06-23T01:19:41.639 INFO:tasks.ceph.osd.2:Restarting daemon 2023-06-23T01:19:41.640 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-06-23T01:19:41.643 INFO:tasks.ceph.osd.2:Started 2023-06-23T01:19:41.644 INFO:tasks.ceph.osd.3:Restarting daemon 2023-06-23T01:19:41.644 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-06-23T01:19:41.647 INFO:tasks.ceph.osd.3:Started 2023-06-23T01:19:41.648 INFO:tasks.ceph.osd.4:Restarting daemon 2023-06-23T01:19:41.649 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-06-23T01:19:41.651 INFO:tasks.ceph.osd.4:Started 2023-06-23T01:19:41.652 INFO:tasks.ceph.osd.5:Restarting daemon 2023-06-23T01:19:41.652 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-06-23T01:19:41.654 INFO:tasks.ceph.osd.5:Started 2023-06-23T01:19:41.654 INFO:tasks.ceph.osd.6:Restarting daemon 2023-06-23T01:19:41.654 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-06-23T01:19:41.656 INFO:tasks.ceph.osd.6:Started 2023-06-23T01:19:41.657 INFO:tasks.ceph.osd.7:Restarting daemon 2023-06-23T01:19:41.657 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-06-23T01:19:41.659 INFO:tasks.ceph.osd.7:Started 2023-06-23T01:19:41.659 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:19:41.683 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:41.679+0000 7fc2582221c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:19:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.808+0000 7f97f8a1f700 1 Processor -- start 2023-06-23T01:19:41.812 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.808+0000 7f97f8a1f700 1 -- start start 2023-06-23T01:19:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.809+0000 7f97f8a1f700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f97f4070220 0x7f97f4070690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.809+0000 7f97f8a1f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f406a480 0x7f97f406a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.809+0000 7f97f8a1f700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97f40c7560 con 0x7f97f40af3a0 2023-06-23T01:19:41.814 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.809+0000 7f97f8a1f700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97f4068d00 con 0x7f97f4070220 2023-06-23T01:19:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.809+0000 7f97f8a1f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f40cb270 con 0x7f97f406a480 2023-06-23T01:19:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.809+0000 7f97f1d9b700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f97f4070220 0x7f97f4070690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f1d9b700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f97f4070220 0x7f97f4070690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59238/0 (socket says 172.21.15.57:59238) 2023-06-23T01:19:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f1d9b700 1 -- 172.21.15.57:0/4236505822 learned_addr learned my addr 172.21.15.57:0/4236505822 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f2d9d700 1 --2- 172.21.15.57:0/4236505822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f406a480 0x7f97f406a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f1d9b700 1 -- 172.21.15.57:0/4236505822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f406a480 msgr2=0x7f97f406a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:41.815 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f1d9b700 1 --2- 172.21.15.57:0/4236505822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f406a480 0x7f97f406a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f1d9b700 1 -- 172.21.15.57:0/4236505822 >> v1:172.21.15.123:6789/0 conn(0x7f97f40af3a0 legacy=0x7f97f4067d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f1d9b700 1 -- 172.21.15.57:0/4236505822 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f4070bd0 con 0x7f97f4070220 2023-06-23T01:19:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f2d9d700 1 --2- 172.21.15.57:0/4236505822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f406a480 0x7f97f406a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:19:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f1d9b700 1 --2- 172.21.15.57:0/4236505822 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f4070690 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f97e40099b0 tx=0x7f97e4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aef3989b39842e73 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f159a700 1 -- 172.21.15.57:0/4236505822 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400b860 con 0x7f97f4070220 2023-06-23T01:19:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f159a700 1 -- 172.21.15.57:0/4236505822 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97e400b9c0 con 0x7f97f4070220 2023-06-23T01:19:41.816 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.810+0000 7f97f159a700 1 -- 172.21.15.57:0/4236505822 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97e400bd60 con 0x7f97f4070220 2023-06-23T01:19:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.811+0000 7f97f8a1f700 1 -- 172.21.15.57:0/4236505822 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 msgr2=0x7f97f4070690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.811+0000 7f97f8a1f700 1 --2- 172.21.15.57:0/4236505822 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f4070690 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f97e40099b0 tx=0x7f97e4009250 comp rx=0 tx=0).stop 2023-06-23T01:19:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.811+0000 7f97f8a1f700 1 -- 172.21.15.57:0/4236505822 shutdown_connections 2023-06-23T01:19:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.811+0000 7f97f8a1f700 1 --2- 172.21.15.57:0/4236505822 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f406a480 0x7f97f406a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.811+0000 7f97f8a1f700 1 --2- 172.21.15.57:0/4236505822 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f4070690 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.811+0000 7f97f8a1f700 1 -- 172.21.15.57:0/4236505822 >> 172.21.15.57:0/4236505822 conn(0x7f97f40c9990 msgr2=0x7f97f40c9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:41.819 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.811+0000 7f97f8a1f700 1 -- 172.21.15.57:0/4236505822 shutdown_connections 2023-06-23T01:19:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.811+0000 7f97f8a1f700 1 -- 172.21.15.57:0/4236505822 wait complete. 2023-06-23T01:19:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.812+0000 7f97f8a1f700 1 Processor -- start 2023-06-23T01:19:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.812+0000 7f97f8a1f700 1 -- start start 2023-06-23T01:19:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f8a1f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f406a480 0x7f97f415eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f8a1f700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f41488a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f8a1f700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f40af3a0 0x7f97f415cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:41.820 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f8a1f700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f415e630 con 0x7f97f4070220 2023-06-23T01:19:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f8a1f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97f415da20 con 0x7f97f406a480 2023-06-23T01:19:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f8a1f700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97f415dd00 con 0x7f97f40af3a0 2023-06-23T01:19:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f1d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f41488a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.821 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f1d9b700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f41488a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59246/0 (socket says 172.21.15.57:59246) 2023-06-23T01:19:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f1d9b700 1 -- 172.21.15.57:0/2010994242 learned_addr learned my addr 172.21.15.57:0/2010994242 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f2d9d700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f40af3a0 0x7f97f415cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f259c700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f406a480 0x7f97f415eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f2d9d700 1 -- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f406a480 msgr2=0x7f97f415eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f2d9d700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f406a480 0x7f97f415eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.822 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f2d9d700 1 -- 172.21.15.57:0/2010994242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 msgr2=0x7f97f41488a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f2d9d700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f41488a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f2d9d700 1 -- 172.21.15.57:0/2010994242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97e40094f0 con 0x7f97f40af3a0 2023-06-23T01:19:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f1d9b700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f41488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.813+0000 7f97f2d9d700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f40af3a0 0x7f97f415cf20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009fa0 tx=0x7f97ec009990 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.814+0000 7f97e27fc700 1 -- 172.21.15.57:0/2010994242 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec00dbb0 con 0x7f97f40af3a0 2023-06-23T01:19:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.814+0000 7f97e27fc700 1 -- 172.21.15.57:0/2010994242 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ec00f040 con 0x7f97f40af3a0 2023-06-23T01:19:41.823 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.814+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97f415dfe0 con 0x7f97f40af3a0 2023-06-23T01:19:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.814+0000 7f97e27fc700 1 -- 172.21.15.57:0/2010994242 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ec015440 con 0x7f97f40af3a0 2023-06-23T01:19:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.814+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97f4157010 con 0x7f97f40af3a0 2023-06-23T01:19:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.814+0000 7f97e27fc700 1 -- 172.21.15.57:0/2010994242 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f97ec015830 con 0x7f97f40af3a0 2023-06-23T01:19:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.814+0000 7f97e27fc700 1 -- 172.21.15.57:0/2010994242 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f97ec015ae0 con 0x7f97f40af3a0 2023-06-23T01:19:41.824 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.815+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97f415e7a0 con 0x7f97f40af3a0 2023-06-23T01:19:41.825 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.817+0000 7f97e27fc700 1 -- 172.21.15.57:0/2010994242 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f97ec01a070 con 0x7f97f40af3a0 2023-06-23T01:19:41.845 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:41.841+0000 7fc2582221c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:19:41.890 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.886+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97f4064b50 con 0x7f97f40af3a0 2023-06-23T01:19:41.891 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.887+0000 7f97e27fc700 1 -- 172.21.15.57:0/2010994242 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f97ec01f690 con 0x7f97f40af3a0 2023-06-23T01:19:41.891 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:41.891 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":10,"fsid":"cbac5275-6376-4f14-97f5-8fe5f66396c8","created":"2023-06-23T01:19:39.130505+0000","modified":"2023-06-23T01:19:41.606376+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:19:41.606376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"bb40d155-cae0-4fda-b7af-ce312c958bbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e2dc7148-fc1a-48a4-8b06-013164077813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d30552e4-8c21-4ff9-baf7-07c7ed117bdf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d22728ed-0946-49b6-871d-ab80ec2089c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a39e2e51-40b1-4c91-b045-f6f030dd6546","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"855f67e8-1b47-4fb7-a8c2-2f5df954a8ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a74edd8e-c135-473e-9394-f214d0405572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26725e66-b047-4e78-afc6-352a8ea70559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:19:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f40af3a0 msgr2=0x7f97f415cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f40af3a0 0x7f97f415cf20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009fa0 tx=0x7f97ec009990 comp rx=0 tx=0).stop 2023-06-23T01:19:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 shutdown_connections 2023-06-23T01:19:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f97f40af3a0 0x7f97f415cf20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f97f406a480 0x7f97f415eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 --2- 172.21.15.57:0/2010994242 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f97f4070220 0x7f97f41488a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:41.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 >> 172.21.15.57:0/2010994242 conn(0x7f97f40c9990 msgr2=0x7f97f4158a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:41.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 shutdown_connections 2023-06-23T01:19:41.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:41.888+0000 7f97f8a1f700 1 -- 172.21.15.57:0/2010994242 wait complete. 2023-06-23T01:19:41.904 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-06-23T01:19:41.904 INFO:tasks.ceph:Waiting for OSDs to come up 2023-06-23T01:19:41.918 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:41.914+0000 7fc2582221c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:19:42.015 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.011+0000 7f8140b431c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:19:42.086 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.082+0000 7f8140b431c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:19:42.103 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:42.099+0000 7fc2582221c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:19:42.183 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:42.180+0000 7fc2582221c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:19:42.218 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.215+0000 7f8140b431c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:19:42.250 INFO:tasks.ceph.osd.4.smithi123.stderr:2023-06-23T01:19:42.246+0000 7fda7bd4a540 -1 Falling back to public interface 2023-06-23T01:19:42.250 INFO:tasks.ceph.osd.5.smithi123.stderr:2023-06-23T01:19:42.246+0000 7f6e01a42540 -1 Falling back to public interface 2023-06-23T01:19:42.258 INFO:tasks.ceph.osd.7.smithi123.stderr:2023-06-23T01:19:42.254+0000 7f5d7193f540 -1 Falling back to public interface 2023-06-23T01:19:42.259 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-06-23T01:19:42.255+0000 7fed1e13b540 -1 Falling back to public interface 2023-06-23T01:19:42.262 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-06-23T01:19:42.258+0000 7f6c1c9e9540 -1 Falling back to public interface 2023-06-23T01:19:42.266 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:42.263+0000 7fc2582221c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:19:42.272 INFO:tasks.ceph.osd.6.smithi123.stderr:2023-06-23T01:19:42.268+0000 7f800d7f2540 -1 Falling back to public interface 2023-06-23T01:19:42.277 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-06-23T01:19:42.273+0000 7fdf2411b540 -1 Falling back to public interface 2023-06-23T01:19:42.278 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-06-23T01:19:42.273+0000 7fbdf8dcb540 -1 Falling back to public interface 2023-06-23T01:19:42.278 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-06-23T01:19:42.274+0000 7fdf1e527700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2023-06-23T01:19:42.288 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.284+0000 7f8140b431c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:19:42.367 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:42.363+0000 7fc2582221c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:19:42.465 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.461+0000 7f8140b431c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:19:42.542 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.538+0000 7f8140b431c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:19:42.553 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:42.549+0000 7fc2582221c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:19:42.618 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.614+0000 7f8140b431c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:19:42.636 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:42.632+0000 7fc2582221c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:19:42.709 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:19:42.710 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.705+0000 7f8140b431c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:19:42.828 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.825+0000 7f7c77856700 1 Processor -- start 2023-06-23T01:19:42.829 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.825+0000 7f7c77856700 1 -- start start 2023-06-23T01:19:42.830 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c77856700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7c700b1f10 0x7f7c70142720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:42.831 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c77856700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c700b58b0 con 0x7f7c700b4b20 2023-06-23T01:19:42.831 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c77856700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c700b5b90 con 0x7f7c700b22e0 2023-06-23T01:19:42.831 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c77856700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c700b56b0 con 0x7f7c700b1f10 2023-06-23T01:19:42.831 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c75df3700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f7c700b4b20 0x7f7c700b4fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:38584/0 (socket says 172.21.15.57:38584) 2023-06-23T01:19:42.831 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c75df3700 1 -- 172.21.15.57:0/2227683370 learned_addr learned my addr 172.21.15.57:0/2227683370 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:42.831 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c755f2700 1 --2- 172.21.15.57:0/2227683370 >> v2:172.21.15.123:3300/0 conn(0x7f7c700b1f10 0x7f7c70142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:42.832 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c67fff700 1 -- 172.21.15.57:0/2227683370 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891819527 0 0) 0x7f7c700b58b0 con 0x7f7c700b4b20 2023-06-23T01:19:42.832 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c67fff700 1 -- 172.21.15.57:0/2227683370 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c5c003570 con 0x7f7c700b4b20 2023-06-23T01:19:42.832 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c755f2700 1 -- 172.21.15.57:0/2227683370 >> v1:172.21.15.123:6789/0 conn(0x7f7c700b22e0 legacy=0x7f7c70142c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:42.832 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c67fff700 1 -- 172.21.15.57:0/2227683370 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2105698970 0 0) 0x7f7c700b5b90 con 0x7f7c700b22e0 2023-06-23T01:19:42.832 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c755f2700 1 -- 172.21.15.57:0/2227683370 >> v1:172.21.15.57:6789/0 conn(0x7f7c700b4b20 legacy=0x7f7c700b4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:42.832 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c755f2700 1 -- 172.21.15.57:0/2227683370 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c700cecf0 con 0x7f7c700b1f10 2023-06-23T01:19:42.832 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.826+0000 7f7c755f2700 1 --2- 172.21.15.57:0/2227683370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b1f10 0x7f7c70142720 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0098e0 tx=0x7f7c6c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f027f2e3dcef6ff server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:42.833 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c67fff700 1 -- 172.21.15.57:0/2227683370 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00b640 con 0x7f7c700b1f10 2023-06-23T01:19:42.833 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c67fff700 1 -- 172.21.15.57:0/2227683370 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c00b7a0 con 0x7f7c700b1f10 2023-06-23T01:19:42.833 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c67fff700 1 -- 172.21.15.57:0/2227683370 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6c00bb40 con 0x7f7c700b1f10 2023-06-23T01:19:42.835 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c77856700 1 -- 172.21.15.57:0/2227683370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b1f10 msgr2=0x7f7c70142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:42.835 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c77856700 1 --2- 172.21.15.57:0/2227683370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b1f10 0x7f7c70142720 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7c6c0098e0 tx=0x7f7c6c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:42.835 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c77856700 1 -- 172.21.15.57:0/2227683370 shutdown_connections 2023-06-23T01:19:42.835 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c77856700 1 --2- 172.21.15.57:0/2227683370 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b1f10 0x7f7c70142720 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:42.836 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c77856700 1 -- 172.21.15.57:0/2227683370 >> 172.21.15.57:0/2227683370 conn(0x7f7c7006d960 msgr2=0x7f7c7006dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:42.836 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c77856700 1 -- 172.21.15.57:0/2227683370 shutdown_connections 2023-06-23T01:19:42.836 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.827+0000 7f7c77856700 1 -- 172.21.15.57:0/2227683370 wait complete. 2023-06-23T01:19:42.836 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.829+0000 7f7c77856700 1 Processor -- start 2023-06-23T01:19:42.836 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.829+0000 7f7c77856700 1 -- start start 2023-06-23T01:19:42.837 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c77856700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c700b1f10 0x7f7c7015b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:42.837 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c77856700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c700b22e0 0x7f7c701587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:42.837 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c77856700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b4b20 0x7f7c70158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:42.837 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c77856700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c70158620 con 0x7f7c700b1f10 2023-06-23T01:19:42.837 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c77856700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7015abc0 con 0x7f7c700b4b20 2023-06-23T01:19:42.837 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c77856700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c7013df80 con 0x7f7c700b22e0 2023-06-23T01:19:42.837 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c755f2700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c700b1f10 0x7f7c7015b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:42.838 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c75df3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b4b20 0x7f7c70158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:42.838 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c74df1700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c700b22e0 0x7f7c701587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:42.838 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c755f2700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c700b1f10 0x7f7c7015b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:59254/0 (socket says 172.21.15.57:59254) 2023-06-23T01:19:42.838 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c755f2700 1 -- 172.21.15.57:0/2728488468 learned_addr learned my addr 172.21.15.57:0/2728488468 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:42.838 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c75df3700 1 -- 172.21.15.57:0/2728488468 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c700b22e0 msgr2=0x7f7c701587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:42.839 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c75df3700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c700b22e0 0x7f7c701587e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:42.839 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c75df3700 1 -- 172.21.15.57:0/2728488468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c700b1f10 msgr2=0x7f7c7015b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:42.839 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c75df3700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c700b1f10 0x7f7c7015b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:42.839 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c75df3700 1 -- 172.21.15.57:0/2728488468 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c6c009420 con 0x7f7c700b4b20 2023-06-23T01:19:42.839 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c74df1700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c700b22e0 0x7f7c701587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:42.839 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c755f2700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c700b1f10 0x7f7c7015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:42.839 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c75df3700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b4b20 0x7f7c70158d20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800cb90 tx=0x7f7c68001980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:42.840 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c657fa700 1 -- 172.21.15.57:0/2728488468 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800a7e0 con 0x7f7c700b4b20 2023-06-23T01:19:42.840 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c657fa700 1 -- 172.21.15.57:0/2728488468 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6800a940 con 0x7f7c700b4b20 2023-06-23T01:19:42.840 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c657fa700 1 -- 172.21.15.57:0/2728488468 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c6800acc0 con 0x7f7c700b4b20 2023-06-23T01:19:42.840 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.830+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7013d6e0 con 0x7f7c700b4b20 2023-06-23T01:19:42.840 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.831+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7013dca0 con 0x7f7c700b4b20 2023-06-23T01:19:42.840 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.831+0000 7f7c657fa700 1 -- 172.21.15.57:0/2728488468 <== mon.1 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f7c68007580 con 0x7f7c700b4b20 2023-06-23T01:19:42.840 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.831+0000 7f7c657fa700 1 -- 172.21.15.57:0/2728488468 <== mon.1 v2:172.21.15.123:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f7c6801cc60 con 0x7f7c700b4b20 2023-06-23T01:19:42.840 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.832+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7015a990 con 0x7f7c700b4b20 2023-06-23T01:19:42.841 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.834+0000 7f7c657fa700 1 -- 172.21.15.57:0/2728488468 <== mon.1 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f7c7015a990 con 0x7f7c700b4b20 2023-06-23T01:19:42.888 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.885+0000 7f8140b431c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:19:42.907 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.903+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c70064b50 con 0x7f7c700b4b20 2023-06-23T01:19:42.908 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.904+0000 7f7c657fa700 1 -- 172.21.15.57:0/2728488468 <== mon.1 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f7c68021060 con 0x7f7c700b4b20 2023-06-23T01:19:42.908 INFO:teuthology.misc.health.smithi057.stdout: 2023-06-23T01:19:42.909 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":10,"fsid":"cbac5275-6376-4f14-97f5-8fe5f66396c8","created":"2023-06-23T01:19:39.130505+0000","modified":"2023-06-23T01:19:41.606376+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:19:41.606376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"bb40d155-cae0-4fda-b7af-ce312c958bbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e2dc7148-fc1a-48a4-8b06-013164077813","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d30552e4-8c21-4ff9-baf7-07c7ed117bdf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"d22728ed-0946-49b6-871d-ab80ec2089c6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"a39e2e51-40b1-4c91-b045-f6f030dd6546","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"855f67e8-1b47-4fb7-a8c2-2f5df954a8ab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"a74edd8e-c135-473e-9394-f214d0405572","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"26725e66-b047-4e78-afc6-352a8ea70559","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:19:42.909 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b4b20 msgr2=0x7f7c70158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:42.911 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b4b20 0x7f7c70158d20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7c6800cb90 tx=0x7f7c68001980 comp rx=0 tx=0).stop 2023-06-23T01:19:42.912 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 shutdown_connections 2023-06-23T01:19:42.912 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7c700b4b20 0x7f7c70158d20 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:42.912 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7c700b1f10 0x7f7c7015b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:42.912 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 --2- 172.21.15.57:0/2728488468 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7c700b22e0 0x7f7c701587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:42.913 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 >> 172.21.15.57:0/2728488468 conn(0x7f7c7006d960 msgr2=0x7f7c700b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:42.913 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 shutdown_connections 2023-06-23T01:19:42.913 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:42.905+0000 7f7c77856700 1 -- 172.21.15.57:0/2728488468 wait complete. 2023-06-23T01:19:42.920 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-23T01:19:42.938 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:42.935+0000 7fc2582221c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:19:42.968 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:42.964+0000 7f8140b431c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:19:43.011 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:43.007+0000 7fc2582221c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:19:43.264 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:43.261+0000 7f8140b431c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:19:43.333 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:43.330+0000 7f8140b431c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:19:43.542 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:43.538+0000 7fc2582221c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:19:43.837 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:43.833+0000 7f8140b431c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:19:43.867 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:43.863+0000 7fc2582221c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:19:44.013 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:44.010+0000 7fc2582221c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:19:44.149 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:44.146+0000 7f8140b431c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:19:44.290 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:44.286+0000 7f8140b431c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:19:44.385 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:44.382+0000 7fc2582221c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:19:44.535 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:44.531+0000 7fc2582221c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:19:44.644 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:44.640+0000 7f8140b431c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:19:44.691 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:44.687+0000 7fc2582221c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:19:44.795 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:44.791+0000 7f8140b431c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:19:44.805 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:44.801+0000 7fc2582221c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:19:44.948 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:44.944+0000 7f8140b431c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:19:45.095 INFO:tasks.ceph.mgr.x.smithi123.stderr:2023-06-23T01:19:45.092+0000 7f8140b431c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:19:45.728 INFO:tasks.ceph.osd.5.smithi123.stderr:2023-06-23T01:19:45.724+0000 7f6e01a42540 -1 osd.5 0 log_to_monitors true 2023-06-23T01:19:45.786 INFO:tasks.ceph.osd.4.smithi123.stderr:2023-06-23T01:19:45.783+0000 7fda7bd4a540 -1 osd.4 0 log_to_monitors true 2023-06-23T01:19:45.787 INFO:tasks.ceph.osd.6.smithi123.stderr:2023-06-23T01:19:45.783+0000 7f800d7f2540 -1 osd.6 0 log_to_monitors true 2023-06-23T01:19:45.797 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-06-23T01:19:45.793+0000 7f6c1c9e9540 -1 osd.1 0 log_to_monitors true 2023-06-23T01:19:45.797 INFO:tasks.ceph.osd.7.smithi123.stderr:2023-06-23T01:19:45.794+0000 7f5d7193f540 -1 osd.7 0 log_to_monitors true 2023-06-23T01:19:45.798 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-06-23T01:19:45.794+0000 7fed1e13b540 -1 osd.0 0 log_to_monitors true 2023-06-23T01:19:45.804 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-06-23T01:19:45.800+0000 7fbdf8dcb540 -1 osd.3 0 log_to_monitors true 2023-06-23T01:19:45.804 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-06-23T01:19:45.801+0000 7fdf2411b540 -1 osd.2 0 log_to_monitors true 2023-06-23T01:19:45.827 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:45.823+0000 7fc21c904700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-06-23T01:19:45.828 INFO:tasks.ceph.mgr.y.smithi057.stderr:2023-06-23T01:19:45.824+0000 7fc21c904700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-06-23T01:19:48.923 INFO:tasks.ceph.osd.7.smithi123.stderr:2023-06-23T01:19:48.920+0000 7f5d68544700 -1 osd.7 0 waiting for initial osdmap 2023-06-23T01:19:48.933 INFO:tasks.ceph.osd.7.smithi123.stderr:2023-06-23T01:19:48.930+0000 7f5d6133b700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:19:48.952 INFO:tasks.ceph.osd.4.smithi123.stderr:2023-06-23T01:19:48.949+0000 7fda7294f700 -1 osd.4 0 waiting for initial osdmap 2023-06-23T01:19:48.958 INFO:tasks.ceph.osd.5.smithi123.stderr:2023-06-23T01:19:48.955+0000 7f6df8647700 -1 osd.5 0 waiting for initial osdmap 2023-06-23T01:19:48.959 INFO:tasks.ceph.osd.4.smithi123.stderr:2023-06-23T01:19:48.955+0000 7fda6b746700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:19:48.964 INFO:tasks.ceph.osd.5.smithi123.stderr:2023-06-23T01:19:48.961+0000 7f6df143e700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:19:48.965 INFO:tasks.ceph.osd.6.smithi123.stderr:2023-06-23T01:19:48.961+0000 7f8005bfa700 -1 osd.6 0 waiting for initial osdmap 2023-06-23T01:19:48.970 INFO:tasks.ceph.osd.6.smithi123.stderr:2023-06-23T01:19:48.966+0000 7f7ffd1ee700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:19:49.114 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-06-23T01:19:49.110+0000 7fbdef9d0700 -1 osd.3 0 waiting for initial osdmap 2023-06-23T01:19:49.121 INFO:tasks.ceph.osd.3.smithi057.stderr:2023-06-23T01:19:49.118+0000 7fbde87c7700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:19:49.178 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-06-23T01:19:49.174+0000 7fed14d40700 -1 osd.0 0 waiting for initial osdmap 2023-06-23T01:19:49.181 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-06-23T01:19:49.177+0000 7fdf1ad20700 -1 osd.2 0 waiting for initial osdmap 2023-06-23T01:19:49.183 INFO:tasks.ceph.osd.0.smithi057.stderr:2023-06-23T01:19:49.180+0000 7fed0db37700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:19:49.190 INFO:tasks.ceph.osd.2.smithi057.stderr:2023-06-23T01:19:49.187+0000 7fdf13b17700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:19:49.204 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-06-23T01:19:49.200+0000 7f6c135ee700 -1 osd.1 0 waiting for initial osdmap 2023-06-23T01:19:49.208 INFO:tasks.ceph.osd.1.smithi057.stderr:2023-06-23T01:19:49.205+0000 7f6c0c3e5700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:19:49.727 DEBUG:teuthology.orchestra.run.smithi057:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:19:49.853 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.850+0000 7f11b1ed0700 1 Processor -- start 2023-06-23T01:19:49.854 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.850+0000 7f11b1ed0700 1 -- start start 2023-06-23T01:19:49.856 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.850+0000 7f11b1ed0700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f11ac0b22e0 0x7f11ac142e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:49.856 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.850+0000 7f11b1ed0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 0x7f11ac0b4f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:49.856 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11b1ed0700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ac0b58b0 con 0x7f11ac0b1f10 2023-06-23T01:19:49.856 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11b1ed0700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ac0b5b90 con 0x7f11ac0b22e0 2023-06-23T01:19:49.856 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11b1ed0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac0cecf0 con 0x7f11ac0b4b20 2023-06-23T01:19:49.856 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aaffd700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f11ac0b22e0 0x7f11ac142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:49.857 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11ab7fe700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f11ac0b1f10 0x7f11ac142720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47134/0 (socket says 172.21.15.57:47134) 2023-06-23T01:19:49.857 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11ab7fe700 1 -- 172.21.15.57:0/2173109697 learned_addr learned my addr 172.21.15.57:0/2173109697 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:49.857 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11abfff700 1 --2- 172.21.15.57:0/2173109697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 0x7f11ac0b4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:49.857 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aaffd700 1 -- 172.21.15.57:0/2173109697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 msgr2=0x7f11ac0b4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:49.857 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aaffd700 1 --2- 172.21.15.57:0/2173109697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 0x7f11ac0b4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:49.857 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aaffd700 1 -- 172.21.15.57:0/2173109697 >> v1:172.21.15.57:6789/0 conn(0x7f11ac0b1f10 legacy=0x7f11ac142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:49.857 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aaffd700 1 -- 172.21.15.57:0/2173109697 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac0cefd0 con 0x7f11ac0b22e0 2023-06-23T01:19:49.858 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aa7fc700 1 -- 172.21.15.57:0/2173109697 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991541088 0 0) 0x7f11ac0b58b0 con 0x7f11ac0b1f10 2023-06-23T01:19:49.858 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aaffd700 1 --2- 172.21.15.57:0/2173109697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac142e30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11a00098e0 tx=0x7f11a00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265b40fec8a3afb5 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:49.858 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aa7fc700 1 -- 172.21.15.57:0/2173109697 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a0009e80 con 0x7f11ac0b22e0 2023-06-23T01:19:49.858 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aa7fc700 1 -- 172.21.15.57:0/2173109697 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a000b860 con 0x7f11ac0b22e0 2023-06-23T01:19:49.858 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.851+0000 7f11aa7fc700 1 -- 172.21.15.57:0/2173109697 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a000bbe0 con 0x7f11ac0b22e0 2023-06-23T01:19:49.858 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.852+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2173109697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 msgr2=0x7f11ac142e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:49.858 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.852+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2173109697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac142e30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f11a00098e0 tx=0x7f11a00091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:49.859 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.852+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2173109697 shutdown_connections 2023-06-23T01:19:49.859 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.852+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2173109697 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 0x7f11ac0b4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:49.859 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.852+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2173109697 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac142e30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:49.859 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.852+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2173109697 >> 172.21.15.57:0/2173109697 conn(0x7f11ac06d960 msgr2=0x7f11ac06dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:49.859 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.852+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2173109697 shutdown_connections 2023-06-23T01:19:49.859 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.852+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2173109697 wait complete. 2023-06-23T01:19:49.861 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.853+0000 7f11b1ed0700 1 Processor -- start 2023-06-23T01:19:49.861 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.853+0000 7f11b1ed0700 1 -- start start 2023-06-23T01:19:49.862 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11b1ed0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ac0b1f10 0x7f11ac158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:49.862 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11b1ed0700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac13c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:49.862 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11b1ed0700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 0x7f11ac13cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:49.862 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11b1ed0700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac1695b0 con 0x7f11ac0b22e0 2023-06-23T01:19:49.862 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11b1ed0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac159210 con 0x7f11ac0b1f10 2023-06-23T01:19:49.862 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11b1ed0700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac13eab0 con 0x7f11ac0b4b20 2023-06-23T01:19:49.862 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11aaffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac13c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:49.862 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11aaffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac13c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42586/0 (socket says 172.21.15.57:42586) 2023-06-23T01:19:49.863 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11aaffd700 1 -- 172.21.15.57:0/2696434988 learned_addr learned my addr 172.21.15.57:0/2696434988 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:49.863 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11aaffd700 1 -- 172.21.15.57:0/2696434988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 msgr2=0x7f11ac13cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:49.863 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11aaffd700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 0x7f11ac13cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:49.863 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11aaffd700 1 -- 172.21.15.57:0/2696434988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ac0b1f10 msgr2=0x7f11ac158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:49.863 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11ab7fe700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ac0b1f10 0x7f11ac158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:49.863 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11aaffd700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ac0b1f10 0x7f11ac158cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:49.863 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.854+0000 7f11aaffd700 1 -- 172.21.15.57:0/2696434988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11a0009420 con 0x7f11ac0b22e0 2023-06-23T01:19:49.864 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.855+0000 7f11aaffd700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac13c750 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11ac089200 tx=0x7f11a0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:49.864 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.855+0000 7f11937fe700 1 -- 172.21.15.57:0/2696434988 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a000bbe0 con 0x7f11ac0b22e0 2023-06-23T01:19:49.864 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.855+0000 7f11937fe700 1 -- 172.21.15.57:0/2696434988 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a0011420 con 0x7f11ac0b22e0 2023-06-23T01:19:49.864 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.855+0000 7f11937fe700 1 -- 172.21.15.57:0/2696434988 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a00117c0 con 0x7f11ac0b22e0 2023-06-23T01:19:49.864 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.855+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac13ed90 con 0x7f11ac0b22e0 2023-06-23T01:19:49.864 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.855+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac13f350 con 0x7f11ac0b22e0 2023-06-23T01:19:49.864 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.856+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac169720 con 0x7f11ac0b22e0 2023-06-23T01:19:49.864 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.856+0000 7f11937fe700 1 -- 172.21.15.57:0/2696434988 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f11a0011bb0 con 0x7f11ac0b22e0 2023-06-23T01:19:49.865 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.857+0000 7f11937fe700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1198066e00 0x7f11980692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:49.865 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.857+0000 7f11937fe700 1 -- 172.21.15.57:0/2696434988 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f11a0084fa0 con 0x7f11ac0b22e0 2023-06-23T01:19:49.865 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.857+0000 7f11ab7fe700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1198066e00 0x7f11980692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:49.865 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.857+0000 7f11ab7fe700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1198066e00 0x7f11980692b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f119c002220 tx=0x7f119c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:49.865 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:49.862+0000 7f11937fe700 1 -- 172.21.15.57:0/2696434988 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f11a0053140 con 0x7f11ac0b22e0 2023-06-23T01:19:50.066 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.062+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11ac064b50 con 0x7f11ac0b22e0 2023-06-23T01:19:50.067 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.063+0000 7f11937fe700 1 -- 172.21.15.57:0/2696434988 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10492 (secure 0 0 0) 0x7f11a0015020 con 0x7f11ac0b22e0 2023-06-23T01:19:50.067 INFO:teuthology.misc.health.smithi057.stdout: 2023-06-23T01:19:50.068 INFO:teuthology.misc.health.smithi057.stdout:{"epoch":13,"fsid":"cbac5275-6376-4f14-97f5-8fe5f66396c8","created":"2023-06-23T01:19:39.130505+0000","modified":"2023-06-23T01:19:49.924346+0000","last_up_change":"2023-06-23T01:19:49.924346+0000","last_in_change":"2023-06-23T01:19:41.606376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"bb40d155-cae0-4fda-b7af-ce312c958bbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":113976},{"type":"v1","addr":"172.21.15.57:6801","nonce":113976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":113976},{"type":"v1","addr":"172.21.15.57:6803","nonce":113976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":113976},{"type":"v1","addr":"172.21.15.57:6807","nonce":113976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":113976},{"type":"v1","addr":"172.21.15.57:6805","nonce":113976}]},"public_addr":"172.21.15.57:6801/113976","cluster_addr":"172.21.15.57:6803/113976","heartbeat_back_addr":"172.21.15.57:6807/113976","heartbeat_front_addr":"172.21.15.57:6805/113976","state":["exists","up"]},{"osd":1,"uuid":"e2dc7148-fc1a-48a4-8b06-013164077813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":113977},{"type":"v1","addr":"172.21.15.57:6809","nonce":113977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":113977},{"type":"v1","addr":"172.21.15.57:6811","nonce":113977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":113977},{"type":"v1","addr":"172.21.15.57:6815","nonce":113977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":113977},{"type":"v1","addr":"172.21.15.57:6813","nonce":113977}]},"public_addr":"172.21.15.57:6809/113977","cluster_addr":"172.21.15.57:6811/113977","heartbeat_back_addr":"172.21.15.57:6815/113977","heartbeat_front_addr":"172.21.15.57:6813/113977","state":["exists","up"]},{"osd":2,"uuid":"d30552e4-8c21-4ff9-baf7-07c7ed117bdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6817","nonce":113978},{"type":"v1","addr":"172.21.15.57:6819","nonce":113978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6821","nonce":113978},{"type":"v1","addr":"172.21.15.57:6823","nonce":113978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6829","nonce":113978},{"type":"v1","addr":"172.21.15.57:6831","nonce":113978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6825","nonce":113978},{"type":"v1","addr":"172.21.15.57:6827","nonce":113978}]},"public_addr":"172.21.15.57:6819/113978","cluster_addr":"172.21.15.57:6823/113978","heartbeat_back_addr":"172.21.15.57:6831/113978","heartbeat_front_addr":"172.21.15.57:6827/113978","state":["exists","up"]},{"osd":3,"uuid":"d22728ed-0946-49b6-871d-ab80ec2089c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":113995},{"type":"v1","addr":"172.21.15.57:6818","nonce":113995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":113995},{"type":"v1","addr":"172.21.15.57:6822","nonce":113995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":113995},{"type":"v1","addr":"172.21.15.57:6830","nonce":113995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":113995},{"type":"v1","addr":"172.21.15.57:6826","nonce":113995}]},"public_addr":"172.21.15.57:6818/113995","cluster_addr":"172.21.15.57:6822/113995","heartbeat_back_addr":"172.21.15.57:6830/113995","heartbeat_front_addr":"172.21.15.57:6826/113995","state":["exists","up"]},{"osd":4,"uuid":"a39e2e51-40b1-4c91-b045-f6f030dd6546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":114171},{"type":"v1","addr":"172.21.15.123:6801","nonce":114171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6803","nonce":114171},{"type":"v1","addr":"172.21.15.123:6805","nonce":114171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6811","nonce":114171},{"type":"v1","addr":"172.21.15.123:6813","nonce":114171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6807","nonce":114171},{"type":"v1","addr":"172.21.15.123:6809","nonce":114171}]},"public_addr":"172.21.15.123:6801/114171","cluster_addr":"172.21.15.123:6805/114171","heartbeat_back_addr":"172.21.15.123:6813/114171","heartbeat_front_addr":"172.21.15.123:6809/114171","state":["exists","up"]},{"osd":5,"uuid":"855f67e8-1b47-4fb7-a8c2-2f5df954a8ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":114172},{"type":"v1","addr":"172.21.15.123:6804","nonce":114172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":114172},{"type":"v1","addr":"172.21.15.123:6808","nonce":114172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":114172},{"type":"v1","addr":"172.21.15.123:6815","nonce":114172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":114172},{"type":"v1","addr":"172.21.15.123:6812","nonce":114172}]},"public_addr":"172.21.15.123:6804/114172","cluster_addr":"172.21.15.123:6808/114172","heartbeat_back_addr":"172.21.15.123:6815/114172","heartbeat_front_addr":"172.21.15.123:6812/114172","state":["exists","up"]},{"osd":6,"uuid":"a74edd8e-c135-473e-9394-f214d0405572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":114216},{"type":"v1","addr":"172.21.15.123:6825","nonce":114216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":114216},{"type":"v1","addr":"172.21.15.123:6827","nonce":114216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":114216},{"type":"v1","addr":"172.21.15.123:6831","nonce":114216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":114216},{"type":"v1","addr":"172.21.15.123:6829","nonce":114216}]},"public_addr":"172.21.15.123:6825/114216","cluster_addr":"172.21.15.123:6827/114216","heartbeat_back_addr":"172.21.15.123:6831/114216","heartbeat_front_addr":"172.21.15.123:6829/114216","state":["exists","up"]},{"osd":7,"uuid":"26725e66-b047-4e78-afc6-352a8ea70559","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":114173},{"type":"v1","addr":"172.21.15.123:6817","nonce":114173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":114173},{"type":"v1","addr":"172.21.15.123:6819","nonce":114173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":114173},{"type":"v1","addr":"172.21.15.123:6823","nonce":114173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":114173},{"type":"v1","addr":"172.21.15.123:6821","nonce":114173}]},"public_addr":"172.21.15.123:6817/114173","cluster_addr":"172.21.15.123:6819/114173","heartbeat_back_addr":"172.21.15.123:6823/114173","heartbeat_front_addr":"172.21.15.123:6821/114173","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:19:50.069 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1198066e00 msgr2=0x7f11980692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:50.069 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1198066e00 0x7f11980692b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f119c002220 tx=0x7f119c009040 comp rx=0 tx=0).stop 2023-06-23T01:19:50.069 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 msgr2=0x7f11ac13c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:50.070 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac13c750 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f11ac089200 tx=0x7f11a0017000 comp rx=0 tx=0).stop 2023-06-23T01:19:50.070 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 shutdown_connections 2023-06-23T01:19:50.070 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1198066e00 0x7f11980692b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.070 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f11ac0b4b20 0x7f11ac13cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.070 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f11ac0b1f10 0x7f11ac158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.070 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 --2- 172.21.15.57:0/2696434988 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f11ac0b22e0 0x7f11ac13c750 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.070 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 >> 172.21.15.57:0/2696434988 conn(0x7f11ac06d960 msgr2=0x7f11ac0b1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:50.071 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 shutdown_connections 2023-06-23T01:19:50.071 INFO:teuthology.misc.health.smithi057.stderr:2023-06-23T01:19:50.065+0000 7f11b1ed0700 1 -- 172.21.15.57:0/2696434988 wait complete. 2023-06-23T01:19:50.079 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-06-23T01:19:50.080 INFO:tasks.ceph:Creating RBD pool 2023-06-23T01:19:50.080 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd pool create rbd 8 2023-06-23T01:19:50.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.198+0000 7f7d09f28700 1 Processor -- start 2023-06-23T01:19:50.202 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.198+0000 7f7d09f28700 1 -- start start 2023-06-23T01:19:50.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d09f28700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7d04153810 0x7f7d040c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:50.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d09f28700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d040c8eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:50.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d09f28700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d040c89d0 con 0x7f7d04159890 2023-06-23T01:19:50.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d09f28700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d04159db0 con 0x7f7d04153810 2023-06-23T01:19:50.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d09f28700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0415a090 con 0x7f7d040ccca0 2023-06-23T01:19:50.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d03fff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d040c8eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:50.204 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d037fe700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f7d04153810 0x7f7d040c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:50.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d03fff700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d040c8eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.57:43050/0 (socket says 172.21.15.57:43050) 2023-06-23T01:19:50.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d03fff700 1 -- 172.21.15.57:0/522864113 learned_addr learned my addr 172.21.15.57:0/522864113 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:50.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d02ffd700 1 --1- >> v1:172.21.15.123:6789/0 conn(0x7f7d04159890 0x7f7d040cc590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.57:39602/0 (socket says 172.21.15.57:39602) 2023-06-23T01:19:50.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.199+0000 7f7d037fe700 1 --2- 172.21.15.57:0/522864113 >> v2:172.21.15.123:3300/0 conn(0x7f7d04153810 0x7f7d040c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.57:43174/0 (socket says 172.21.15.57:43174) 2023-06-23T01:19:50.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d027fc700 1 -- 172.21.15.57:0/522864113 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277957511 0 0) 0x7f7d040c89d0 con 0x7f7d04159890 2023-06-23T01:19:50.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d027fc700 1 -- 172.21.15.57:0/522864113 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cf0003570 con 0x7f7d04159890 2023-06-23T01:19:50.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d03fff700 1 -- 172.21.15.57:0/522864113 >> v2:172.21.15.123:3300/0 conn(0x7f7d04153810 msgr2=0x7f7d040c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d03fff700 1 --2- 172.21.15.57:0/522864113 >> v2:172.21.15.123:3300/0 conn(0x7f7d04153810 0x7f7d040c6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d03fff700 1 -- 172.21.15.57:0/522864113 >> v1:172.21.15.123:6789/0 conn(0x7f7d04159890 legacy=0x7f7d040cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d03fff700 1 -- 172.21.15.57:0/522864113 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0415a370 con 0x7f7d040ccca0 2023-06-23T01:19:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d037fe700 1 --2- 172.21.15.57:0/522864113 >> v2:172.21.15.123:3300/0 conn(0x7f7d04153810 0x7f7d040c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d03fff700 1 --2- 172.21.15.57:0/522864113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d040c8eb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009f20 tx=0x7f7cfc009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b657bc739ace5c1d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d027fc700 1 -- 172.21.15.57:0/522864113 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc00e070 con 0x7f7d040ccca0 2023-06-23T01:19:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d027fc700 1 -- 172.21.15.57:0/522864113 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cfc00b970 con 0x7f7d040ccca0 2023-06-23T01:19:50.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d027fc700 1 -- 172.21.15.57:0/522864113 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cfc0092e0 con 0x7f7d040ccca0 2023-06-23T01:19:50.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d09f28700 1 -- 172.21.15.57:0/522864113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 msgr2=0x7f7d040c8eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:50.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.200+0000 7f7d09f28700 1 --2- 172.21.15.57:0/522864113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d040c8eb0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7cfc009f20 tx=0x7f7cfc009840 comp rx=0 tx=0).stop 2023-06-23T01:19:50.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.201+0000 7f7d09f28700 1 -- 172.21.15.57:0/522864113 shutdown_connections 2023-06-23T01:19:50.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.201+0000 7f7d09f28700 1 --2- 172.21.15.57:0/522864113 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d040c8eb0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.201+0000 7f7d09f28700 1 --2- 172.21.15.57:0/522864113 >> v2:172.21.15.123:3300/0 conn(0x7f7d04153810 0x7f7d040c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.201+0000 7f7d09f28700 1 -- 172.21.15.57:0/522864113 >> 172.21.15.57:0/522864113 conn(0x7f7d04070220 msgr2=0x7f7d04070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:50.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.201+0000 7f7d09f28700 1 -- 172.21.15.57:0/522864113 shutdown_connections 2023-06-23T01:19:50.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.201+0000 7f7d09f28700 1 -- 172.21.15.57:0/522864113 wait complete. 2023-06-23T01:19:50.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.202+0000 7f7d09f28700 1 Processor -- start 2023-06-23T01:19:50.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.202+0000 7f7d09f28700 1 -- start start 2023-06-23T01:19:50.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d09f28700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d04144ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:50.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d09f28700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d04153810 0x7f7d040c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:50.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d09f28700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d04159890 0x7f7d04151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:50.210 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d09f28700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c5a30 con 0x7f7d04159890 2023-06-23T01:19:50.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d09f28700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c4e20 con 0x7f7d04153810 2023-06-23T01:19:50.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d09f28700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c5100 con 0x7f7d040ccca0 2023-06-23T01:19:50.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d03fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d04159890 0x7f7d04151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:50.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d037fe700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d04144ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:50.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d02ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d04153810 0x7f7d040c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:50.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d03fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d04159890 0x7f7d04151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42600/0 (socket says 172.21.15.57:42600) 2023-06-23T01:19:50.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d03fff700 1 -- 172.21.15.57:0/2793380800 learned_addr learned my addr 172.21.15.57:0/2793380800 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:50.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d037fe700 1 -- 172.21.15.57:0/2793380800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d04153810 msgr2=0x7f7d040c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:50.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d037fe700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d04153810 0x7f7d040c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d037fe700 1 -- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d04159890 msgr2=0x7f7d04151700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:50.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d037fe700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d04159890 0x7f7d04151700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:50.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d037fe700 1 -- 172.21.15.57:0/2793380800 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4009420 con 0x7f7d040ccca0 2023-06-23T01:19:50.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d03fff700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d04159890 0x7f7d04151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:50.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d037fe700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d04144ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7cf400ec80 tx=0x7f7cf40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:50.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7cfb7fe700 1 -- 172.21.15.57:0/2793380800 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf400c850 con 0x7f7d040ccca0 2023-06-23T01:19:50.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7cfb7fe700 1 -- 172.21.15.57:0/2793380800 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf400c9b0 con 0x7f7d040ccca0 2023-06-23T01:19:50.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cfc009ae0 con 0x7f7d040ccca0 2023-06-23T01:19:50.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.203+0000 7f7cfb7fe700 1 -- 172.21.15.57:0/2793380800 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf400cd70 con 0x7f7d040ccca0 2023-06-23T01:19:50.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.204+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d040c56c0 con 0x7f7d040ccca0 2023-06-23T01:19:50.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.204+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d04064b50 con 0x7f7d040ccca0 2023-06-23T01:19:50.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.205+0000 7f7cfb7fe700 1 -- 172.21.15.57:0/2793380800 <== mon.2 v2:172.21.15.123:3301/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f7cf4013680 con 0x7f7d040ccca0 2023-06-23T01:19:50.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.205+0000 7f7cfb7fe700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f7ce4066e00 0x7f7ce40692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:50.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.205+0000 7f7cfb7fe700 1 -- 172.21.15.57:0/2793380800 <== mon.2 v2:172.21.15.123:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4467+0+0 (secure 0 0 0) 0x7f7cf4084080 con 0x7f7d040ccca0 2023-06-23T01:19:50.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.205+0000 7f7d02ffd700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f7ce4066e00 0x7f7ce40692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:50.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.206+0000 7f7d02ffd700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f7ce4066e00 0x7f7ce40692b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d040c77a0 tx=0x7f7cec007d40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:50.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.211+0000 7f7cfb7fe700 1 -- 172.21.15.57:0/2793380800 <== mon.2 v2:172.21.15.123:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f7cf4051960 con 0x7f7d040ccca0 2023-06-23T01:19:50.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:50.409+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f7d040c5240 con 0x7f7d040ccca0 2023-06-23T01:19:51.231 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.227+0000 7f7cfb7fe700 1 -- 172.21.15.57:0/2793380800 <== mon.2 v2:172.21.15.123:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f7d040c5240 con 0x7f7d040ccca0 2023-06-23T01:19:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f7ce4066e00 msgr2=0x7f7ce40692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f7ce4066e00 0x7f7ce40692b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d040c77a0 tx=0x7f7cec007d40 comp rx=0 tx=0).stop 2023-06-23T01:19:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 msgr2=0x7f7d04144ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d04144ae0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7cf400ec80 tx=0x7f7cf40091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 shutdown_connections 2023-06-23T01:19:51.235 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f7ce4066e00 0x7f7ce40692b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f7d04159890 0x7f7d04151700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f7d040ccca0 0x7f7d04144ae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 --2- 172.21.15.57:0/2793380800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d04153810 0x7f7d040c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 >> 172.21.15.57:0/2793380800 conn(0x7f7d04070220 msgr2=0x7f7d04152ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 shutdown_connections 2023-06-23T01:19:51.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.231+0000 7f7d09f28700 1 -- 172.21.15.57:0/2793380800 wait complete. 2023-06-23T01:19:51.237 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'rbd' created 2023-06-23T01:19:51.249 DEBUG:teuthology.orchestra.run.smithi057:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-06-23T01:19:51.378 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.375+0000 7f520d2a7700 1 Processor -- start 2023-06-23T01:19:51.379 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.375+0000 7f520d2a7700 1 -- start start 2023-06-23T01:19:51.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.375+0000 7f520d2a7700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f52080b4d80 0x7f52080b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:51.383 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f520d2a7700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52080b5320 con 0x7f5208159310 2023-06-23T01:19:51.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f520d2a7700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52080b5600 con 0x7f52081596e0 2023-06-23T01:19:51.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f520d2a7700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52080b58e0 con 0x7f52080b4d80 2023-06-23T01:19:51.384 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5206ffd700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f5208159310 0x7f52080b43b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47142/0 (socket says 172.21.15.57:47142) 2023-06-23T01:19:51.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5206ffd700 1 -- 172.21.15.57:0/3208007504 learned_addr learned my addr 172.21.15.57:0/3208007504 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:51.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f52067fc700 1 --2- 172.21.15.57:0/3208007504 >> v2:172.21.15.123:3300/0 conn(0x7f52080b4d80 0x7f52080b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:51.385 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836278295 0 0) 0x7f52080b5320 con 0x7f5208159310 2023-06-23T01:19:51.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f0003570 con 0x7f5208159310 2023-06-23T01:19:51.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 454776705 0 0) 0x7f52080b5600 con 0x7f52081596e0 2023-06-23T01:19:51.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52080b5320 con 0x7f52081596e0 2023-06-23T01:19:51.386 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1092817339 0 0) 0x7f51f0003570 con 0x7f5208159310 2023-06-23T01:19:51.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 >> v2:172.21.15.123:3300/0 conn(0x7f52080b4d80 msgr2=0x7f52080b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:51.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 --2- 172.21.15.57:0/3208007504 >> v2:172.21.15.123:3300/0 conn(0x7f52080b4d80 0x7f52080b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 >> v1:172.21.15.123:6789/0 conn(0x7f52081596e0 legacy=0x7f52080ce780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:51.387 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52080b5bc0 con 0x7f5208159310 2023-06-23T01:19:51.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3756962719 0 0) 0x7f51f8001470 con 0x7f5208159310 2023-06-23T01:19:51.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 >> v1:172.21.15.57:6789/0 conn(0x7f5208159310 legacy=0x7f52080b43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:51.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f0003bf0 0x7f51f0003fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:51.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f0004810 0x7f51f0004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:51.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f51f000ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:51.388 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52080b58e0 con 0x7f51f00051c0 2023-06-23T01:19:51.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f000c0b0 con 0x7f51f0003bf0 2023-06-23T01:19:51.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.376+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f000c270 con 0x7f51f0004810 2023-06-23T01:19:51.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f5206ffd700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f0003bf0 0x7f51f0003fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:51.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52077fe700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f51f000ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:51.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52067fc700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f0004810 0x7f51f0004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:51.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52077fe700 1 -- 172.21.15.57:0/3208007504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f0004810 msgr2=0x7f51f0004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:51.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52077fe700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f51f0004810 0x7f51f0004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.389 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52077fe700 1 -- 172.21.15.57:0/3208007504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f0003bf0 msgr2=0x7f51f0003fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:51.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52077fe700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f51f0003bf0 0x7f51f0003fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52077fe700 1 -- 172.21.15.57:0/3208007504 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f0003570 con 0x7f51f00051c0 2023-06-23T01:19:51.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52077fe700 1 -- 172.21.15.57:0/3208007504 reap_dead start 2023-06-23T01:19:51.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f52077fe700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f51f000ba60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5200002120 tx=0x7f52000017a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23141056e829a3c8 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:51.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5200001ab0 con 0x7f51f00051c0 2023-06-23T01:19:51.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f520000c980 con 0x7f51f00051c0 2023-06-23T01:19:51.390 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.377+0000 7f5205ffb700 1 -- 172.21.15.57:0/3208007504 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f520000cd00 con 0x7f51f00051c0 2023-06-23T01:19:51.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.378+0000 7f520d2a7700 1 -- 172.21.15.57:0/3208007504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 msgr2=0x7f51f000ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:51.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.378+0000 7f520d2a7700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f51f000ba60 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5200002120 tx=0x7f52000017a0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.378+0000 7f520d2a7700 1 -- 172.21.15.57:0/3208007504 shutdown_connections 2023-06-23T01:19:51.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.378+0000 7f520d2a7700 1 --2- 172.21.15.57:0/3208007504 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f51f000ba60 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.378+0000 7f520d2a7700 1 -- 172.21.15.57:0/3208007504 >> 172.21.15.57:0/3208007504 conn(0x7f5208067340 msgr2=0x7f5208067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:51.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.378+0000 7f520d2a7700 1 -- 172.21.15.57:0/3208007504 shutdown_connections 2023-06-23T01:19:51.391 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.378+0000 7f520d2a7700 1 -- 172.21.15.57:0/3208007504 wait complete. 2023-06-23T01:19:51.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.379+0000 7f520d2a7700 1 Processor -- start 2023-06-23T01:19:51.392 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.379+0000 7f520d2a7700 1 -- start start 2023-06-23T01:19:51.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f520d2a7700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f5208151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:51.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f520d2a7700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f520816a760 0x7f520813a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:51.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f520d2a7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f520814e850 0x7f520815aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:51.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f520d2a7700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f520814f560 con 0x7f51f00051c0 2023-06-23T01:19:51.394 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f520d2a7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5208150a40 con 0x7f520814e850 2023-06-23T01:19:51.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f520d2a7700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52081512c0 con 0x7f520816a760 2023-06-23T01:19:51.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f5208151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:51.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f5208151750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42632/0 (socket says 172.21.15.57:42632) 2023-06-23T01:19:51.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 -- 172.21.15.57:0/3428974080 learned_addr learned my addr 172.21.15.57:0/3428974080 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:51.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f52077fe700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f520814e850 0x7f520815aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:51.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f52067fc700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f520816a760 0x7f520813a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:51.395 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 -- 172.21.15.57:0/3428974080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f520816a760 msgr2=0x7f520813a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:51.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f520816a760 0x7f520813a770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 -- 172.21.15.57:0/3428974080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f520814e850 msgr2=0x7f520815aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:51.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f520814e850 0x7f520815aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:51.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 -- 172.21.15.57:0/3428974080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f520814f280 con 0x7f51f00051c0 2023-06-23T01:19:51.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.380+0000 7f5206ffd700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f5208151750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f51f8001ed0 tx=0x7f52080b4b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:51.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.381+0000 7f51eeffd700 1 -- 172.21.15.57:0/3428974080 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8011590 con 0x7f51f00051c0 2023-06-23T01:19:51.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.381+0000 7f51eeffd700 1 -- 172.21.15.57:0/3428974080 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51f8011710 con 0x7f51f00051c0 2023-06-23T01:19:51.396 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.381+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5208150c70 con 0x7f51f00051c0 2023-06-23T01:19:51.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.381+0000 7f51eeffd700 1 -- 172.21.15.57:0/3428974080 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f8011ad0 con 0x7f51f00051c0 2023-06-23T01:19:51.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.381+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f520814ecd0 con 0x7f51f00051c0 2023-06-23T01:19:51.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.382+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5208064b50 con 0x7f51f00051c0 2023-06-23T01:19:51.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.382+0000 7f51eeffd700 1 -- 172.21.15.57:0/3428974080 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f51f801d460 con 0x7f51f00051c0 2023-06-23T01:19:51.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.382+0000 7f51eeffd700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f51f4066e00 0x7f51f40692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:51.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.383+0000 7f52067fc700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f51f4066e00 0x7f51f40692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:51.397 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.383+0000 7f51eeffd700 1 -- 172.21.15.57:0/3428974080 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 5184+0+0 (secure 0 0 0) 0x7f51f8084230 con 0x7f51f00051c0 2023-06-23T01:19:51.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.383+0000 7f52067fc700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f51f4066e00 0x7f51f40692b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f51fc006ed0 tx=0x7f51fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:51.398 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.387+0000 7f51eeffd700 1 -- 172.21.15.57:0/3428974080 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f51f8051850 con 0x7f51f00051c0 2023-06-23T01:19:51.583 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:51.580+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f52080b33f0 con 0x7f51f00051c0 2023-06-23T01:19:52.232 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.228+0000 7f51eeffd700 1 -- 172.21.15.57:0/3428974080 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7f51f8054e70 con 0x7f51f00051c0 2023-06-23T01:19:52.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f51f4066e00 msgr2=0x7f51f40692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f51f4066e00 0x7f51f40692b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f51fc006ed0 tx=0x7f51fc008040 comp rx=0 tx=0).stop 2023-06-23T01:19:52.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 msgr2=0x7f5208151750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.236 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f5208151750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f51f8001ed0 tx=0x7f52080b4b00 comp rx=0 tx=0).stop 2023-06-23T01:19:52.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 shutdown_connections 2023-06-23T01:19:52.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f51f4066e00 0x7f51f40692b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f520814e850 0x7f520815aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f51f00051c0 0x7f5208151750 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 --2- 172.21.15.57:0/3428974080 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f520816a760 0x7f520813a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 >> 172.21.15.57:0/3428974080 conn(0x7f5208067340 msgr2=0x7f52080cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:52.237 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 shutdown_connections 2023-06-23T01:19:52.238 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.232+0000 7f520d2a7700 1 -- 172.21.15.57:0/3428974080 wait complete. 2023-06-23T01:19:52.238 INFO:teuthology.orchestra.run.smithi057.stderr:enabled application 'rbd' on pool 'rbd' 2023-06-23T01:19:52.252 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-06-23T01:19:52.252 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-23T01:19:52.252 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-23T01:19:52.254 INFO:tasks.ceph.mds.a:Started 2023-06-23T01:19:52.255 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-23T01:19:52.255 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-23T01:19:52.257 INFO:tasks.ceph.mds.c:Started 2023-06-23T01:19:52.257 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-23T01:19:52.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-23T01:19:52.261 INFO:tasks.ceph.mds.b:Started 2023-06-23T01:19:52.262 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-23T01:19:52.263 DEBUG:teuthology.orchestra.run.smithi123:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-23T01:19:52.265 INFO:tasks.ceph.mds.d:Started 2023-06-23T01:19:52.266 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-06-23T01:19:52.266 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:19:52.340 INFO:tasks.ceph.mds.c.smithi057.stdout:starting mds.c at 2023-06-23T01:19:52.341 INFO:tasks.ceph.mds.a.smithi057.stdout:starting mds.a at 2023-06-23T01:19:52.346 INFO:tasks.ceph.mds.b.smithi123.stdout:starting mds.b at 2023-06-23T01:19:52.350 INFO:tasks.ceph.mds.d.smithi123.stdout:starting mds.d at 2023-06-23T01:19:52.401 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.397+0000 7f903d4bb700 1 Processor -- start 2023-06-23T01:19:52.402 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.398+0000 7f903d4bb700 1 -- start start 2023-06-23T01:19:52.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.398+0000 7f903d4bb700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f90380af810 0x7f903815ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.398+0000 7f903d4bb700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903815ac70 con 0x7f90380af440 2023-06-23T01:19:52.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.398+0000 7f903d4bb700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f903815b7f0 con 0x7f90380cd7d0 2023-06-23T01:19:52.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.398+0000 7f903d4bb700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f903815aec0 con 0x7f90380af810 2023-06-23T01:19:52.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.398+0000 7f9036ffd700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f90380af440 0x7f90380cb200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47164/0 (socket says 172.21.15.57:47164) 2023-06-23T01:19:52.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.398+0000 7f9036ffd700 1 -- 172.21.15.57:0/1206468511 learned_addr learned my addr 172.21.15.57:0/1206468511 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:52.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f90367fc700 1 --2- 172.21.15.57:0/1206468511 >> v2:172.21.15.123:3300/0 conn(0x7f90380af810 0x7f903815ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f9035ffb700 1 -- 172.21.15.57:0/1206468511 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044940946 0 0) 0x7f903815ac70 con 0x7f90380af440 2023-06-23T01:19:52.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f9035ffb700 1 -- 172.21.15.57:0/1206468511 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9020003570 con 0x7f90380af440 2023-06-23T01:19:52.411 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f9035ffb700 1 -- 172.21.15.57:0/1206468511 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3545600720 0 0) 0x7f903815b7f0 con 0x7f90380cd7d0 2023-06-23T01:19:52.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f9035ffb700 1 -- 172.21.15.57:0/1206468511 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f903815ac70 con 0x7f90380cd7d0 2023-06-23T01:19:52.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f90367fc700 1 -- 172.21.15.57:0/1206468511 >> v1:172.21.15.123:6789/0 conn(0x7f90380cd7d0 legacy=0x7f903815bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f90367fc700 1 -- 172.21.15.57:0/1206468511 >> v1:172.21.15.57:6789/0 conn(0x7f90380af440 legacy=0x7f90380cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f90367fc700 1 -- 172.21.15.57:0/1206468511 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903815b1a0 con 0x7f90380af810 2023-06-23T01:19:52.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f90367fc700 1 --2- 172.21.15.57:0/1206468511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af810 0x7f903815ce90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f90280098e0 tx=0x7f90280091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9226458fd4e4e00 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:52.412 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f9035ffb700 1 -- 172.21.15.57:0/1206468511 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028011630 con 0x7f90380af810 2023-06-23T01:19:52.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f9035ffb700 1 -- 172.21.15.57:0/1206468511 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9028011790 con 0x7f90380af810 2023-06-23T01:19:52.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.399+0000 7f9035ffb700 1 -- 172.21.15.57:0/1206468511 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9028011b30 con 0x7f90380af810 2023-06-23T01:19:52.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.400+0000 7f903d4bb700 1 -- 172.21.15.57:0/1206468511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af810 msgr2=0x7f903815ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.400+0000 7f903d4bb700 1 --2- 172.21.15.57:0/1206468511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af810 0x7f903815ce90 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f90280098e0 tx=0x7f90280091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.400+0000 7f903d4bb700 1 -- 172.21.15.57:0/1206468511 shutdown_connections 2023-06-23T01:19:52.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.400+0000 7f903d4bb700 1 --2- 172.21.15.57:0/1206468511 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af810 0x7f903815ce90 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.400+0000 7f903d4bb700 1 -- 172.21.15.57:0/1206468511 >> 172.21.15.57:0/1206468511 conn(0x7f9038067340 msgr2=0x7f9038067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:52.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.400+0000 7f903d4bb700 1 -- 172.21.15.57:0/1206468511 shutdown_connections 2023-06-23T01:19:52.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.400+0000 7f903d4bb700 1 -- 172.21.15.57:0/1206468511 wait complete. 2023-06-23T01:19:52.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.401+0000 7f903d4bb700 1 Processor -- start 2023-06-23T01:19:52.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.401+0000 7f903d4bb700 1 -- start start 2023-06-23T01:19:52.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f903d4bb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af440 0x7f9038150ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f903d4bb700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90380af810 0x7f90380b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f903d4bb700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90380cd7d0 0x7f90380b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f903d4bb700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90380afde0 con 0x7f90380cd7d0 2023-06-23T01:19:52.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f903d4bb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90380b00c0 con 0x7f90380af440 2023-06-23T01:19:52.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f903d4bb700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90380b03a0 con 0x7f90380af810 2023-06-23T01:19:52.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90380cd7d0 0x7f90380b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90380cd7d0 0x7f90380b40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42678/0 (socket says 172.21.15.57:42678) 2023-06-23T01:19:52.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 -- 172.21.15.57:0/1370603863 learned_addr learned my addr 172.21.15.57:0/1370603863 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:52.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90367fc700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90380af810 0x7f90380b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f9036ffd700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af440 0x7f9038150ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 -- 172.21.15.57:0/1370603863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90380af810 msgr2=0x7f90380b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90380af810 0x7f90380b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 -- 172.21.15.57:0/1370603863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af440 msgr2=0x7f9038150ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af440 0x7f9038150ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 -- 172.21.15.57:0/1370603863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9028009420 con 0x7f90380cd7d0 2023-06-23T01:19:52.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.402+0000 7f90377fe700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90380cd7d0 0x7f90380b40e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f903000c910 tx=0x7f90300027a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:52.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.403+0000 7f901effd700 1 -- 172.21.15.57:0/1370603863 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903000a7e0 con 0x7f90380cd7d0 2023-06-23T01:19:52.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.403+0000 7f901effd700 1 -- 172.21.15.57:0/1370603863 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903000a960 con 0x7f90380cd7d0 2023-06-23T01:19:52.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.403+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90380b0680 con 0x7f90380cd7d0 2023-06-23T01:19:52.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.403+0000 7f901effd700 1 -- 172.21.15.57:0/1370603863 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f903000ad20 con 0x7f90380cd7d0 2023-06-23T01:19:52.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.403+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90380b0c40 con 0x7f90380cd7d0 2023-06-23T01:19:52.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.404+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90380b0e30 con 0x7f90380cd7d0 2023-06-23T01:19:52.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.404+0000 7f901effd700 1 -- 172.21.15.57:0/1370603863 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f9030012630 con 0x7f90380cd7d0 2023-06-23T01:19:52.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.405+0000 7f901effd700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f9024066e80 0x7f9024069330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.405+0000 7f901effd700 1 -- 172.21.15.57:0/1370603863 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9030084610 con 0x7f90380cd7d0 2023-06-23T01:19:52.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.405+0000 7f9036ffd700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f9024066e80 0x7f9024069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.405+0000 7f9036ffd700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f9024066e80 0x7f9024069330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f902c0058d0 tx=0x7f902c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:52.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.409+0000 7f901effd700 1 -- 172.21.15.57:0/1370603863 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f9030051c20 con 0x7f90380cd7d0 2023-06-23T01:19:52.607 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.603+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9038064b50 con 0x7f90380cd7d0 2023-06-23T01:19:52.608 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.604+0000 7f901effd700 1 -- 172.21.15.57:0/1370603863 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+13770 (secure 0 0 0) 0x7f9030055240 con 0x7f90380cd7d0 2023-06-23T01:19:52.608 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:52.609 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":15,"fsid":"cbac5275-6376-4f14-97f5-8fe5f66396c8","created":"2023-06-23T01:19:39.130505+0000","modified":"2023-06-23T01:19:52.222395+0000","last_up_change":"2023-06-23T01:19:49.924346+0000","last_in_change":"2023-06-23T01:19:41.606376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T01:19:50.441526+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T01:19:50.876930+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"bb40d155-cae0-4fda-b7af-ce312c958bbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":113976},{"type":"v1","addr":"172.21.15.57:6801","nonce":113976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":113976},{"type":"v1","addr":"172.21.15.57:6803","nonce":113976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":113976},{"type":"v1","addr":"172.21.15.57:6807","nonce":113976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":113976},{"type":"v1","addr":"172.21.15.57:6805","nonce":113976}]},"public_addr":"172.21.15.57:6801/113976","cluster_addr":"172.21.15.57:6803/113976","heartbeat_back_addr":"172.21.15.57:6807/113976","heartbeat_front_addr":"172.21.15.57:6805/113976","state":["exists","up"]},{"osd":1,"uuid":"e2dc7148-fc1a-48a4-8b06-013164077813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":113977},{"type":"v1","addr":"172.21.15.57:6809","nonce":113977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":113977},{"type":"v1","addr":"172.21.15.57:6811","nonce":113977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":113977},{"type":"v1","addr":"172.21.15.57:6815","nonce":113977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":113977},{"type":"v1","addr":"172.21.15.57:6813","nonce":113977}]},"public_addr":"172.21.15.57:6809/113977","cluster_addr":"172.21.15.57:6811/113977","heartbeat_back_addr":"172.21.15.57:6815/113977","heartbeat_front_addr":"172.21.15.57:6813/113977","state":["exists","up"]},{"osd":2,"uuid":"d30552e4-8c21-4ff9-baf7-07c7ed117bdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6817","nonce":113978},{"type":"v1","addr":"172.21.15.57:6819","nonce":113978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6821","nonce":113978},{"type":"v1","addr":"172.21.15.57:6823","nonce":113978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6829","nonce":113978},{"type":"v1","addr":"172.21.15.57:6831","nonce":113978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6825","nonce":113978},{"type":"v1","addr":"172.21.15.57:6827","nonce":113978}]},"public_addr":"172.21.15.57:6819/113978","cluster_addr":"172.21.15.57:6823/113978","heartbeat_back_addr":"172.21.15.57:6831/113978","heartbeat_front_addr":"172.21.15.57:6827/113978","state":["exists","up"]},{"osd":3,"uuid":"d22728ed-0946-49b6-871d-ab80ec2089c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":113995},{"type":"v1","addr":"172.21.15.57:6818","nonce":113995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":113995},{"type":"v1","addr":"172.21.15.57:6822","nonce":113995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":113995},{"type":"v1","addr":"172.21.15.57:6830","nonce":113995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":113995},{"type":"v1","addr":"172.21.15.57:6826","nonce":113995}]},"public_addr":"172.21.15.57:6818/113995","cluster_addr":"172.21.15.57:6822/113995","heartbeat_back_addr":"172.21.15.57:6830/113995","heartbeat_front_addr":"172.21.15.57:6826/113995","state":["exists","up"]},{"osd":4,"uuid":"a39e2e51-40b1-4c91-b045-f6f030dd6546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":114171},{"type":"v1","addr":"172.21.15.123:6801","nonce":114171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6803","nonce":114171},{"type":"v1","addr":"172.21.15.123:6805","nonce":114171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6811","nonce":114171},{"type":"v1","addr":"172.21.15.123:6813","nonce":114171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6807","nonce":114171},{"type":"v1","addr":"172.21.15.123:6809","nonce":114171}]},"public_addr":"172.21.15.123:6801/114171","cluster_addr":"172.21.15.123:6805/114171","heartbeat_back_addr":"172.21.15.123:6813/114171","heartbeat_front_addr":"172.21.15.123:6809/114171","state":["exists","up"]},{"osd":5,"uuid":"855f67e8-1b47-4fb7-a8c2-2f5df954a8ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":114172},{"type":"v1","addr":"172.21.15.123:6804","nonce":114172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":114172},{"type":"v1","addr":"172.21.15.123:6808","nonce":114172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":114172},{"type":"v1","addr":"172.21.15.123:6815","nonce":114172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":114172},{"type":"v1","addr":"172.21.15.123:6812","nonce":114172}]},"public_addr":"172.21.15.123:6804/114172","cluster_addr":"172.21.15.123:6808/114172","heartbeat_back_addr":"172.21.15.123:6815/114172","heartbeat_front_addr":"172.21.15.123:6812/114172","state":["exists","up"]},{"osd":6,"uuid":"a74edd8e-c135-473e-9394-f214d0405572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":114216},{"type":"v1","addr":"172.21.15.123:6825","nonce":114216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":114216},{"type":"v1","addr":"172.21.15.123:6827","nonce":114216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":114216},{"type":"v1","addr":"172.21.15.123:6831","nonce":114216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":114216},{"type":"v1","addr":"172.21.15.123:6829","nonce":114216}]},"public_addr":"172.21.15.123:6825/114216","cluster_addr":"172.21.15.123:6827/114216","heartbeat_back_addr":"172.21.15.123:6831/114216","heartbeat_front_addr":"172.21.15.123:6829/114216","state":["exists","up"]},{"osd":7,"uuid":"26725e66-b047-4e78-afc6-352a8ea70559","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":114173},{"type":"v1","addr":"172.21.15.123:6817","nonce":114173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":114173},{"type":"v1","addr":"172.21.15.123:6819","nonce":114173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":114173},{"type":"v1","addr":"172.21.15.123:6823","nonce":114173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":114173},{"type":"v1","addr":"172.21.15.123:6821","nonce":114173}]},"public_addr":"172.21.15.123:6817/114173","cluster_addr":"172.21.15.123:6819/114173","heartbeat_back_addr":"172.21.15.123:6823/114173","heartbeat_front_addr":"172.21.15.123:6821/114173","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.786276+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.749336+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.794142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.824374+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.812739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.708934+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.806842+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.766984+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:19:52.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f9024066e80 msgr2=0x7f9024069330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f9024066e80 0x7f9024069330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f902c0058d0 tx=0x7f902c009040 comp rx=0 tx=0).stop 2023-06-23T01:19:52.611 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90380cd7d0 msgr2=0x7f90380b40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90380cd7d0 0x7f90380b40e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f903000c910 tx=0x7f90300027a0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 shutdown_connections 2023-06-23T01:19:52.612 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f9024066e80 0x7f9024069330 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f90380cd7d0 0x7f90380b40e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90380af440 0x7f9038150ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 --2- 172.21.15.57:0/1370603863 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f90380af810 0x7f90380b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.613 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 >> 172.21.15.57:0/1370603863 conn(0x7f9038067340 msgr2=0x7f90381588d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:52.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 shutdown_connections 2023-06-23T01:19:52.614 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.606+0000 7f903d4bb700 1 -- 172.21.15.57:0/1370603863 wait complete. 2023-06-23T01:19:52.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-06-23T01:19:50.441526+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:19:50.876930+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-23T01:19:52.623 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T01:19:52.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.756+0000 7f853dfa3700 1 Processor -- start 2023-06-23T01:19:52.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.756+0000 7f853dfa3700 1 -- start start 2023-06-23T01:19:52.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f853dfa3700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8538156fe0 0x7f853806c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f853dfa3700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8538159860 con 0x7f853815ce60 2023-06-23T01:19:52.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f853dfa3700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853815d3b0 con 0x7f8538156c10 2023-06-23T01:19:52.761 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f853dfa3700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8538069410 con 0x7f8538156fe0 2023-06-23T01:19:52.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f8536ffd700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8538156fe0 0x7f853806c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f8536ffd700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8538156fe0 0x7f853806c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42680/0 (socket says 172.21.15.57:42680) 2023-06-23T01:19:52.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f8536ffd700 1 -- 172.21.15.57:0/2964131593 learned_addr learned my addr 172.21.15.57:0/2964131593 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:52.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f8536ffd700 1 -- 172.21.15.57:0/2964131593 >> v1:172.21.15.123:6789/0 conn(0x7f8538156c10 legacy=0x7f85380c8610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f8536ffd700 1 -- 172.21.15.57:0/2964131593 >> v1:172.21.15.57:6789/0 conn(0x7f853815ce60 legacy=0x7f853806b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f85367fc700 1 -- 172.21.15.57:0/2964131593 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997475691 0 0) 0x7f8538159860 con 0x7f853815ce60 2023-06-23T01:19:52.762 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.757+0000 7f8536ffd700 1 -- 172.21.15.57:0/2964131593 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853815d690 con 0x7f8538156fe0 2023-06-23T01:19:52.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.758+0000 7f8536ffd700 1 --2- 172.21.15.57:0/2964131593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156fe0 0x7f853806c800 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f852c0099b0 tx=0x7f852c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d132ac9d6706ded server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:52.763 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.758+0000 7f85367fc700 1 -- 172.21.15.57:0/2964131593 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00b860 con 0x7f8538156fe0 2023-06-23T01:19:52.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.758+0000 7f85367fc700 1 -- 172.21.15.57:0/2964131593 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852c00b9c0 con 0x7f8538156fe0 2023-06-23T01:19:52.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.758+0000 7f85367fc700 1 -- 172.21.15.57:0/2964131593 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852c00bd60 con 0x7f8538156fe0 2023-06-23T01:19:52.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.758+0000 7f853dfa3700 1 -- 172.21.15.57:0/2964131593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156fe0 msgr2=0x7f853806c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.764 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.758+0000 7f853dfa3700 1 --2- 172.21.15.57:0/2964131593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156fe0 0x7f853806c800 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f852c0099b0 tx=0x7f852c009250 comp rx=0 tx=0).stop 2023-06-23T01:19:52.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.759+0000 7f853dfa3700 1 -- 172.21.15.57:0/2964131593 shutdown_connections 2023-06-23T01:19:52.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.759+0000 7f853dfa3700 1 --2- 172.21.15.57:0/2964131593 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156fe0 0x7f853806c800 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.759+0000 7f853dfa3700 1 -- 172.21.15.57:0/2964131593 >> 172.21.15.57:0/2964131593 conn(0x7f85380cdfc0 msgr2=0x7f85380ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:52.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.759+0000 7f853dfa3700 1 -- 172.21.15.57:0/2964131593 shutdown_connections 2023-06-23T01:19:52.765 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.759+0000 7f853dfa3700 1 -- 172.21.15.57:0/2964131593 wait complete. 2023-06-23T01:19:52.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.760+0000 7f853dfa3700 1 Processor -- start 2023-06-23T01:19:52.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.760+0000 7f853dfa3700 1 -- start start 2023-06-23T01:19:52.767 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f853dfa3700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156c10 0x7f85380b1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f853dfa3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8538156fe0 0x7f853815a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f853dfa3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815ce60 0x7f853815a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f853dfa3700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853815bb60 con 0x7f8538156c10 2023-06-23T01:19:52.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f853dfa3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85380c2690 con 0x7f853815ce60 2023-06-23T01:19:52.768 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f853dfa3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8538169a30 con 0x7f8538156fe0 2023-06-23T01:19:52.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f85377fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156c10 0x7f85380b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f85377fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156c10 0x7f85380b1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42694/0 (socket says 172.21.15.57:42694) 2023-06-23T01:19:52.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.761+0000 7f85377fe700 1 -- 172.21.15.57:0/1768689363 learned_addr learned my addr 172.21.15.57:0/1768689363 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:52.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f8536ffd700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8538156fe0 0x7f853815a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f8537fff700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815ce60 0x7f853815a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f85377fe700 1 -- 172.21.15.57:0/1768689363 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8538156fe0 msgr2=0x7f853815a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.769 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f85377fe700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8538156fe0 0x7f853815a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f85377fe700 1 -- 172.21.15.57:0/1768689363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815ce60 msgr2=0x7f853815a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f85377fe700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815ce60 0x7f853815a5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f85377fe700 1 -- 172.21.15.57:0/1768689363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f852c0094f0 con 0x7f8538156c10 2023-06-23T01:19:52.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f85377fe700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156c10 0x7f85380b1720 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f852800c590 tx=0x7f852800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:52.770 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f851f7fe700 1 -- 172.21.15.57:0/1768689363 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528011590 con 0x7f8538156c10 2023-06-23T01:19:52.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f851f7fe700 1 -- 172.21.15.57:0/1768689363 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8528011710 con 0x7f8538156c10 2023-06-23T01:19:52.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853815b2c0 con 0x7f8538156c10 2023-06-23T01:19:52.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f851f7fe700 1 -- 172.21.15.57:0/1768689363 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8528011ad0 con 0x7f8538156c10 2023-06-23T01:19:52.771 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.762+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f853815b880 con 0x7f8538156c10 2023-06-23T01:19:52.772 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.764+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85380c2460 con 0x7f8538156c10 2023-06-23T01:19:52.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.764+0000 7f851f7fe700 1 -- 172.21.15.57:0/1768689363 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f852801b3b0 con 0x7f8538156c10 2023-06-23T01:19:52.773 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.764+0000 7f851f7fe700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8524066e00 0x7f85240692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:52.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.764+0000 7f851f7fe700 1 -- 172.21.15.57:0/1768689363 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f85280834e0 con 0x7f8538156c10 2023-06-23T01:19:52.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.764+0000 7f8536ffd700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8524066e00 0x7f85240692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:52.774 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.765+0000 7f8536ffd700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8524066e00 0x7f85240692b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f852c0099b0 tx=0x7f852c017000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:52.775 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.770+0000 7f851f7fe700 1 -- 172.21.15.57:0/1768689363 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f85380c2460 con 0x7f8538156c10 2023-06-23T01:19:52.973 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.969+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f85380c2460 con 0x7f8538156c10 2023-06-23T01:19:52.974 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.970+0000 7f851f7fe700 1 -- 172.21.15.57:0/1768689363 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7f85380c2460 con 0x7f8538156c10 2023-06-23T01:19:52.974 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 8 2023-06-23T01:19:52.975 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8524066e00 msgr2=0x7f85240692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8524066e00 0x7f85240692b0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f852c0099b0 tx=0x7f852c017000 comp rx=0 tx=0).stop 2023-06-23T01:19:52.976 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156c10 msgr2=0x7f85380b1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:52.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156c10 0x7f85380b1720 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f852800c590 tx=0x7f852800c8a0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 shutdown_connections 2023-06-23T01:19:52.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8524066e00 0x7f85240692b0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f853815ce60 0x7f853815a5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8538156c10 0x7f85380b1720 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 --2- 172.21.15.57:0/1768689363 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8538156fe0 0x7f853815a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:52.977 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 >> 172.21.15.57:0/1768689363 conn(0x7f85380cdfc0 msgr2=0x7f8538153f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:52.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 shutdown_connections 2023-06-23T01:19:52.978 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:52.972+0000 7f853dfa3700 1 -- 172.21.15.57:0/1768689363 wait complete. 2023-06-23T01:19:52.988 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:19:53.115 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.111+0000 7f83a2aac700 1 Processor -- start 2023-06-23T01:19:53.116 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.112+0000 7f83a2aac700 1 -- start start 2023-06-23T01:19:53.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.112+0000 7f83a2aac700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c153810 0x7f839c0c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.117 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.112+0000 7f83a2aac700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c0c89d0 con 0x7f839c0ccca0 2023-06-23T01:19:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.112+0000 7f83a2aac700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f839c159db0 con 0x7f839c159890 2023-06-23T01:19:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.112+0000 7f83a2aac700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c15a090 con 0x7f839c153810 2023-06-23T01:19:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f83a1049700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f839c0ccca0 0x7f839c0c8eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47190/0 (socket says 172.21.15.57:47190) 2023-06-23T01:19:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f83a1049700 1 -- 172.21.15.57:0/3364344143 learned_addr learned my addr 172.21.15.57:0/3364344143 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f83a0848700 1 --2- 172.21.15.57:0/3364344143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c153810 0x7f839c0c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f839b7fe700 1 -- 172.21.15.57:0/3364344143 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104945393 0 0) 0x7f839c0c89d0 con 0x7f839c0ccca0 2023-06-23T01:19:53.118 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f839b7fe700 1 -- 172.21.15.57:0/3364344143 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f838c003570 con 0x7f839c0ccca0 2023-06-23T01:19:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f839b7fe700 1 -- 172.21.15.57:0/3364344143 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241944653 0 0) 0x7f839c159db0 con 0x7f839c159890 2023-06-23T01:19:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f839b7fe700 1 -- 172.21.15.57:0/3364344143 --> v1:172.21.15.123:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f839c0c89d0 con 0x7f839c159890 2023-06-23T01:19:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f83a0848700 1 -- 172.21.15.57:0/3364344143 >> v1:172.21.15.123:6789/0 conn(0x7f839c159890 legacy=0x7f839c0cc590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f83a0848700 1 -- 172.21.15.57:0/3364344143 >> v1:172.21.15.57:6789/0 conn(0x7f839c0ccca0 legacy=0x7f839c0c8eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.119 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f83a0848700 1 -- 172.21.15.57:0/3364344143 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f839c15a370 con 0x7f839c153810 2023-06-23T01:19:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f83a0848700 1 --2- 172.21.15.57:0/3364344143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c153810 0x7f839c0c6d60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8390009910 tx=0x7f83900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eee256c15fd5df9b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.113+0000 7f839b7fe700 1 -- 172.21.15.57:0/3364344143 <== mon.2 v2:172.21.15.123:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390011630 con 0x7f839c153810 2023-06-23T01:19:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f839b7fe700 1 -- 172.21.15.57:0/3364344143 <== mon.2 v2:172.21.15.123:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390011790 con 0x7f839c153810 2023-06-23T01:19:53.120 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f839b7fe700 1 -- 172.21.15.57:0/3364344143 <== mon.2 v2:172.21.15.123:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390011b30 con 0x7f839c153810 2023-06-23T01:19:53.122 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f83a2aac700 1 -- 172.21.15.57:0/3364344143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c153810 msgr2=0x7f839c0c6d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f83a2aac700 1 --2- 172.21.15.57:0/3364344143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c153810 0x7f839c0c6d60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8390009910 tx=0x7f83900091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f83a2aac700 1 -- 172.21.15.57:0/3364344143 shutdown_connections 2023-06-23T01:19:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f83a2aac700 1 --2- 172.21.15.57:0/3364344143 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c153810 0x7f839c0c6d60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f83a2aac700 1 -- 172.21.15.57:0/3364344143 >> 172.21.15.57:0/3364344143 conn(0x7f839c070220 msgr2=0x7f839c070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f83a2aac700 1 -- 172.21.15.57:0/3364344143 shutdown_connections 2023-06-23T01:19:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.114+0000 7f83a2aac700 1 -- 172.21.15.57:0/3364344143 wait complete. 2023-06-23T01:19:53.123 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.115+0000 7f83a2aac700 1 Processor -- start 2023-06-23T01:19:53.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a2aac700 1 -- start start 2023-06-23T01:19:53.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a2aac700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f839c0ccca0 0x7f839c0c2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a2aac700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f839c153810 0x7f839c15ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a2aac700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c159890 0x7f839c1527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a2aac700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0c1790 con 0x7f839c0ccca0 2023-06-23T01:19:53.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a2aac700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0c1a70 con 0x7f839c153810 2023-06-23T01:19:53.124 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a2aac700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f839c0c1d50 con 0x7f839c159890 2023-06-23T01:19:53.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a0848700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f839c0ccca0 0x7f839c0c2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a0848700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f839c0ccca0 0x7f839c0c2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42716/0 (socket says 172.21.15.57:42716) 2023-06-23T01:19:53.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.116+0000 7f83a0848700 1 -- 172.21.15.57:0/1792782485 learned_addr learned my addr 172.21.15.57:0/1792782485 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:53.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f839bfff700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f839c153810 0x7f839c15ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a1049700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c159890 0x7f839c1527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a0848700 1 -- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c159890 msgr2=0x7f839c1527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.125 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a0848700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c159890 0x7f839c1527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a0848700 1 -- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f839c153810 msgr2=0x7f839c15ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a0848700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f839c153810 0x7f839c15ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a0848700 1 -- 172.21.15.57:0/1792782485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8390009420 con 0x7f839c0ccca0 2023-06-23T01:19:53.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f839bfff700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f839c153810 0x7f839c15ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:19:53.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a0848700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f839c0ccca0 0x7f839c0c2550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8390009f00 tx=0x7f8390017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f8398ff9700 1 -- 172.21.15.57:0/1792782485 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f839001f070 con 0x7f839c0ccca0 2023-06-23T01:19:53.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f8398ff9700 1 -- 172.21.15.57:0/1792782485 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8390000bc0 con 0x7f839c0ccca0 2023-06-23T01:19:53.126 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f839c0c2030 con 0x7f839c0ccca0 2023-06-23T01:19:53.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f8398ff9700 1 -- 172.21.15.57:0/1792782485 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8390025420 con 0x7f839c0ccca0 2023-06-23T01:19:53.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.117+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f839c15b370 con 0x7f839c0ccca0 2023-06-23T01:19:53.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.118+0000 7f8398ff9700 1 -- 172.21.15.57:0/1792782485 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f8390023070 con 0x7f839c0ccca0 2023-06-23T01:19:53.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.119+0000 7f8398ff9700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8380066e00 0x7f83800692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.119+0000 7f8398ff9700 1 -- 172.21.15.57:0/1792782485 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8390085490 con 0x7f839c0ccca0 2023-06-23T01:19:53.127 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.119+0000 7f839bfff700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8380066e00 0x7f83800692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.119+0000 7f839bfff700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8380066e00 0x7f83800692b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f839c15abb0 tx=0x7f8388009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.120+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f839c0c18d0 con 0x7f839c0ccca0 2023-06-23T01:19:53.128 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.125+0000 7f8398ff9700 1 -- 172.21.15.57:0/1792782485 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f839c0c18d0 con 0x7f839c0ccca0 2023-06-23T01:19:53.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.327+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f839c15b590 con 0x7f839c0ccca0 2023-06-23T01:19:53.331 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.327+0000 7f8398ff9700 1 -- 172.21.15.57:0/1792782485 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f839002a0a0 con 0x7f839c0ccca0 2023-06-23T01:19:53.331 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-06-23T01:19:53.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8380066e00 msgr2=0x7f83800692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.333 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8380066e00 0x7f83800692b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f839c15abb0 tx=0x7f8388009040 comp rx=0 tx=0).stop 2023-06-23T01:19:53.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f839c0ccca0 msgr2=0x7f839c0c2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f839c0ccca0 0x7f839c0c2550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8390009f00 tx=0x7f8390017000 comp rx=0 tx=0).stop 2023-06-23T01:19:53.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 shutdown_connections 2023-06-23T01:19:53.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8380066e00 0x7f83800692b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f839c159890 0x7f839c1527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f839c0ccca0 0x7f839c0c2550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.334 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 --2- 172.21.15.57:0/1792782485 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f839c153810 0x7f839c15ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.329+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 >> 172.21.15.57:0/1792782485 conn(0x7f839c070220 msgr2=0x7f839c069dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:53.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.330+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 shutdown_connections 2023-06-23T01:19:53.335 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.330+0000 7f83a2aac700 1 -- 172.21.15.57:0/1792782485 wait complete. 2023-06-23T01:19:53.345 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:19:53.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.467+0000 7f6f9db6c700 1 Processor -- start 2023-06-23T01:19:53.471 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.467+0000 7f6f9db6c700 1 -- start start 2023-06-23T01:19:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f9db6c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 0x7f6f981593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.474 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f9db6c700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6f980cb210 0x7f6f9815ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f9db6c700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f980d0190 con 0x7f6f980cf0c0 2023-06-23T01:19:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f9db6c700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f980cf580 con 0x7f6f980cb210 2023-06-23T01:19:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f9db6c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980cf860 con 0x7f6f98158ff0 2023-06-23T01:19:53.475 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f97fff700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f6f980cb210 0x7f6f9815ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f96ffd700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f6f980cf0c0 0x7f6f980d0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47196/0 (socket says 172.21.15.57:47196) 2023-06-23T01:19:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f96ffd700 1 -- 172.21.15.57:0/2615403226 learned_addr learned my addr 172.21.15.57:0/2615403226 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f977fe700 1 --2- 172.21.15.57:0/2615403226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 0x7f6f981593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.476 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f97fff700 1 -- 172.21.15.57:0/2615403226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 msgr2=0x7f6f981593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f97fff700 1 --2- 172.21.15.57:0/2615403226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 0x7f6f981593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f97fff700 1 -- 172.21.15.57:0/2615403226 >> v1:172.21.15.57:6789/0 conn(0x7f6f980cf0c0 legacy=0x7f6f980d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f97fff700 1 -- 172.21.15.57:0/2615403226 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f980cfb40 con 0x7f6f980cb210 2023-06-23T01:19:53.477 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f967fc700 1 -- 172.21.15.57:0/2615403226 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622781782 0 0) 0x7f6f980d0190 con 0x7f6f980cf0c0 2023-06-23T01:19:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f97fff700 1 --2- 172.21.15.57:0/2615403226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9815ce90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009e60 tx=0x7f6f88009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a955a217c54ba1 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f967fc700 1 -- 172.21.15.57:0/2615403226 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8800e070 con 0x7f6f980cb210 2023-06-23T01:19:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f967fc700 1 -- 172.21.15.57:0/2615403226 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f8800bb60 con 0x7f6f980cb210 2023-06-23T01:19:53.478 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.468+0000 7f6f967fc700 1 -- 172.21.15.57:0/2615403226 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f88012430 con 0x7f6f980cb210 2023-06-23T01:19:53.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.469+0000 7f6f9db6c700 1 -- 172.21.15.57:0/2615403226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 msgr2=0x7f6f9815ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.469+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/2615403226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9815ce90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f6f88009e60 tx=0x7f6f88009780 comp rx=0 tx=0).stop 2023-06-23T01:19:53.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.469+0000 7f6f9db6c700 1 -- 172.21.15.57:0/2615403226 shutdown_connections 2023-06-23T01:19:53.479 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.469+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/2615403226 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9815ce90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.469+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/2615403226 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 0x7f6f981593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.469+0000 7f6f9db6c700 1 -- 172.21.15.57:0/2615403226 >> 172.21.15.57:0/2615403226 conn(0x7f6f98067340 msgr2=0x7f6f98067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:53.480 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.469+0000 7f6f9db6c700 1 -- 172.21.15.57:0/2615403226 shutdown_connections 2023-06-23T01:19:53.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.469+0000 7f6f9db6c700 1 -- 172.21.15.57:0/2615403226 wait complete. 2023-06-23T01:19:53.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.470+0000 7f6f9db6c700 1 Processor -- start 2023-06-23T01:19:53.481 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.470+0000 7f6f9db6c700 1 -- start start 2023-06-23T01:19:53.483 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f9db6c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9816ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f9db6c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f980cf0c0 0x7f6f98156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f9db6c700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 0x7f6f980b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f9db6c700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980b3760 con 0x7f6f980cb210 2023-06-23T01:19:53.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f9db6c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f980b3130 con 0x7f6f980cf0c0 2023-06-23T01:19:53.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f9db6c700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9816a600 con 0x7f6f98158ff0 2023-06-23T01:19:53.484 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f977fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9816ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f977fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9816ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42738/0 (socket says 172.21.15.57:42738) 2023-06-23T01:19:53.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f977fe700 1 -- 172.21.15.57:0/1445653347 learned_addr learned my addr 172.21.15.57:0/1445653347 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:53.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f96ffd700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f980cf0c0 0x7f6f98156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.485 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f97fff700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 0x7f6f980b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f977fe700 1 -- 172.21.15.57:0/1445653347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 msgr2=0x7f6f980b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f977fe700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 0x7f6f980b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.486 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f977fe700 1 -- 172.21.15.57:0/1445653347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f980cf0c0 msgr2=0x7f6f98156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f977fe700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f980cf0c0 0x7f6f98156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.471+0000 7f6f977fe700 1 -- 172.21.15.57:0/1445653347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f88009a20 con 0x7f6f980cb210 2023-06-23T01:19:53.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.472+0000 7f6f977fe700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9816ab00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000e930 tx=0x7f6f900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.487 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.472+0000 7f6f8f7fe700 1 -- 172.21.15.57:0/1445653347 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f90013680 con 0x7f6f980cb210 2023-06-23T01:19:53.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.472+0000 7f6f8f7fe700 1 -- 172.21.15.57:0/1445653347 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f900137e0 con 0x7f6f980cb210 2023-06-23T01:19:53.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.472+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f980b33f0 con 0x7f6f980cb210 2023-06-23T01:19:53.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.472+0000 7f6f8f7fe700 1 -- 172.21.15.57:0/1445653347 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f90013b80 con 0x7f6f980cb210 2023-06-23T01:19:53.488 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.472+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f980b5120 con 0x7f6f980cb210 2023-06-23T01:19:53.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.473+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f98064b50 con 0x7f6f980cb210 2023-06-23T01:19:53.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.473+0000 7f6f8f7fe700 1 -- 172.21.15.57:0/1445653347 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f6f9000c3e0 con 0x7f6f980cb210 2023-06-23T01:19:53.489 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.474+0000 7f6f8f7fe700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f6f78066e90 0x7f6f78069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.474+0000 7f6f8f7fe700 1 -- 172.21.15.57:0/1445653347 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f6f900842d0 con 0x7f6f980cb210 2023-06-23T01:19:53.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.474+0000 7f6f96ffd700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f6f78066e90 0x7f6f78069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.474+0000 7f6f96ffd700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f6f78066e90 0x7f6f78069340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6f980af720 tx=0x7f6f80007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.490 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.478+0000 7f6f8f7fe700 1 -- 172.21.15.57:0/1445653347 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f6f900517f0 con 0x7f6f980cb210 2023-06-23T01:19:53.722 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.719+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f980cfe20 con 0x7f6f980cb210 2023-06-23T01:19:53.723 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.719+0000 7f6f8f7fe700 1 -- 172.21.15.57:0/1445653347 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3524 (secure 0 0 0) 0x7f6f90017de0 con 0x7f6f980cb210 2023-06-23T01:19:53.723 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:53.723 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6833/706528450","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6832","nonce":706528450},{"type":"v1","addr":"172.21.15.123:6833","nonce":706528450}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4238,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.123:6835/4000159665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6834","nonce":4000159665},{"type":"v1","addr":"172.21.15.123:6835","nonce":4000159665}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4251,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6836/4251543665","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6834","nonce":4251543665},{"type":"v1","addr":"172.21.15.57:6836","nonce":4251543665}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4254,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.57:6837/262526719","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6835","nonce":262526719},{"type":"v1","addr":"172.21.15.57:6837","nonce":262526719}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-23T01:19:53.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f6f78066e90 msgr2=0x7f6f78069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f6f78066e90 0x7f6f78069340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f6f980af720 tx=0x7f6f80007d30 comp rx=0 tx=0).stop 2023-06-23T01:19:53.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 msgr2=0x7f6f9816ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.725 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9816ab00 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6f9000e930 tx=0x7f6f900091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 shutdown_connections 2023-06-23T01:19:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f6f78066e90 0x7f6f78069340 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f6f98158ff0 0x7f6f980b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f6f980cb210 0x7f6f9816ab00 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 --2- 172.21.15.57:0/1445653347 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f980cf0c0 0x7f6f98156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 >> 172.21.15.57:0/1445653347 conn(0x7f6f98067340 msgr2=0x7f6f9815cde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 shutdown_connections 2023-06-23T01:19:53.726 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.721+0000 7f6f9db6c700 1 -- 172.21.15.57:0/1445653347 wait complete. 2023-06-23T01:19:53.727 INFO:teuthology.orchestra.run.smithi057.stderr:dumped fsmap epoch 3 2023-06-23T01:19:53.737 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:19:53.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.857+0000 7f8af9731700 1 Processor -- start 2023-06-23T01:19:53.861 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.857+0000 7f8af9731700 1 -- start start 2023-06-23T01:19:53.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af9731700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f8af40b18e0 0x7f8af414f910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af9731700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8af40b1cb0 0x7f8af4159770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af9731700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af40c9200 con 0x7f8af40c8d70 2023-06-23T01:19:53.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af9731700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af406ed10 con 0x7f8af40b1cb0 2023-06-23T01:19:53.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af9731700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8af40c94e0 con 0x7f8af40b18e0 2023-06-23T01:19:53.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af37fe700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8af40b1cb0 0x7f8af4159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af37fe700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f8af40b1cb0 0x7f8af4159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42766/0 (socket says 172.21.15.57:42766) 2023-06-23T01:19:53.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af37fe700 1 -- 172.21.15.57:0/4007391947 learned_addr learned my addr 172.21.15.57:0/4007391947 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:53.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.858+0000 7f8af2ffd700 1 --2- 172.21.15.57:0/4007391947 >> v2:172.21.15.123:3300/0 conn(0x7f8af40b18e0 0x7f8af414f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af37fe700 1 -- 172.21.15.57:0/4007391947 >> v2:172.21.15.123:3300/0 conn(0x7f8af40b18e0 msgr2=0x7f8af414f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af37fe700 1 --2- 172.21.15.57:0/4007391947 >> v2:172.21.15.123:3300/0 conn(0x7f8af40b18e0 0x7f8af414f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af37fe700 1 -- 172.21.15.57:0/4007391947 >> v1:172.21.15.123:6789/0 conn(0x7f8af40c8d70 legacy=0x7f8af414fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af37fe700 1 -- 172.21.15.57:0/4007391947 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af40c97c0 con 0x7f8af40b1cb0 2023-06-23T01:19:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af37fe700 1 --2- 172.21.15.57:0/4007391947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af4159770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8af40b26a0 tx=0x7f8aec009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec02567b6af1f66c server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af1ffb700 1 -- 172.21.15.57:0/4007391947 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec00e070 con 0x7f8af40b1cb0 2023-06-23T01:19:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af1ffb700 1 -- 172.21.15.57:0/4007391947 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aec00ba50 con 0x7f8af40b1cb0 2023-06-23T01:19:53.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af1ffb700 1 -- 172.21.15.57:0/4007391947 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aec0092e0 con 0x7f8af40b1cb0 2023-06-23T01:19:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af9731700 1 -- 172.21.15.57:0/4007391947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 msgr2=0x7f8af4159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af9731700 1 --2- 172.21.15.57:0/4007391947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af4159770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8af40b26a0 tx=0x7f8aec009840 comp rx=0 tx=0).stop 2023-06-23T01:19:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.859+0000 7f8af9731700 1 -- 172.21.15.57:0/4007391947 shutdown_connections 2023-06-23T01:19:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.860+0000 7f8af9731700 1 --2- 172.21.15.57:0/4007391947 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af4159770 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.860+0000 7f8af9731700 1 --2- 172.21.15.57:0/4007391947 >> v2:172.21.15.123:3300/0 conn(0x7f8af40b18e0 0x7f8af414f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.860+0000 7f8af9731700 1 -- 172.21.15.57:0/4007391947 >> 172.21.15.57:0/4007391947 conn(0x7f8af4070220 msgr2=0x7f8af4070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.860+0000 7f8af9731700 1 -- 172.21.15.57:0/4007391947 shutdown_connections 2023-06-23T01:19:53.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.860+0000 7f8af9731700 1 -- 172.21.15.57:0/4007391947 wait complete. 2023-06-23T01:19:53.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.861+0000 7f8af9731700 1 Processor -- start 2023-06-23T01:19:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.861+0000 7f8af9731700 1 -- start start 2023-06-23T01:19:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af9731700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8af40b18e0 0x7f8af414ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af9731700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af416ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af9731700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8af40c8d70 0x7f8af414f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af9731700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af415c0b0 con 0x7f8af40b1cb0 2023-06-23T01:19:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af9731700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af4156a10 con 0x7f8af40c8d70 2023-06-23T01:19:53.869 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af9731700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8af415bdb0 con 0x7f8af40b18e0 2023-06-23T01:19:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af416ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af416ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42778/0 (socket says 172.21.15.57:42778) 2023-06-23T01:19:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 -- 172.21.15.57:0/3738950677 learned_addr learned my addr 172.21.15.57:0/3738950677 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af37fe700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8af40c8d70 0x7f8af414f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af2ffd700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8af40b18e0 0x7f8af414ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 -- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8af40b18e0 msgr2=0x7f8af414ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8af40b18e0 0x7f8af414ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 -- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8af40c8d70 msgr2=0x7f8af414f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8af40c8d70 0x7f8af414f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 -- 172.21.15.57:0/3738950677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aec009ae0 con 0x7f8af40b1cb0 2023-06-23T01:19:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af2ffd700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8af40b18e0 0x7f8af414ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:19:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af37fe700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8af40c8d70 0x7f8af414f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:19:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af27fc700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af416ca20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400c510 tx=0x7f8ae400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8aeaffd700 1 -- 172.21.15.57:0/3738950677 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4011590 con 0x7f8af40b1cb0 2023-06-23T01:19:53.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8aeaffd700 1 -- 172.21.15.57:0/3738950677 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ae4011710 con 0x7f8af40b1cb0 2023-06-23T01:19:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8aeaffd700 1 -- 172.21.15.57:0/3738950677 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ae4011ad0 con 0x7f8af40b1cb0 2023-06-23T01:19:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af416d270 con 0x7f8af40b1cb0 2023-06-23T01:19:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.862+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af416cf60 con 0x7f8af40b1cb0 2023-06-23T01:19:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.863+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af415c260 con 0x7f8af40b1cb0 2023-06-23T01:19:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.864+0000 7f8aeaffd700 1 -- 172.21.15.57:0/3738950677 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f8ae401b3b0 con 0x7f8af40b1cb0 2023-06-23T01:19:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.864+0000 7f8aeaffd700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8ad4066e90 0x7f8ad4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.865+0000 7f8af2ffd700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8ad4066e90 0x7f8ad4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:53.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.865+0000 7f8aeaffd700 1 -- 172.21.15.57:0/3738950677 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f8ae4083a10 con 0x7f8af40b1cb0 2023-06-23T01:19:53.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.865+0000 7f8af2ffd700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8ad4066e90 0x7f8ad4069340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8adc006fd0 tx=0x7f8adc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:53.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:53.868+0000 7f8aeaffd700 1 -- 172.21.15.57:0/3738950677 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f8ae4050f10 con 0x7f8af40b1cb0 2023-06-23T01:19:54.067 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.064+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8af415c260 con 0x7f8af40b1cb0 2023-06-23T01:19:54.068 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.065+0000 7f8aeaffd700 1 -- 172.21.15.57:0/3738950677 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+13744 (secure 0 0 0) 0x7f8af415c260 con 0x7f8af40b1cb0 2023-06-23T01:19:54.069 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:54.069 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":16,"fsid":"cbac5275-6376-4f14-97f5-8fe5f66396c8","created":"2023-06-23T01:19:39.130505+0000","modified":"2023-06-23T01:19:53.227910+0000","last_up_change":"2023-06-23T01:19:49.924346+0000","last_in_change":"2023-06-23T01:19:41.606376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T01:19:50.441526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T01:19:50.876930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"bb40d155-cae0-4fda-b7af-ce312c958bbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":113976},{"type":"v1","addr":"172.21.15.57:6801","nonce":113976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":113976},{"type":"v1","addr":"172.21.15.57:6803","nonce":113976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":113976},{"type":"v1","addr":"172.21.15.57:6807","nonce":113976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":113976},{"type":"v1","addr":"172.21.15.57:6805","nonce":113976}]},"public_addr":"172.21.15.57:6801/113976","cluster_addr":"172.21.15.57:6803/113976","heartbeat_back_addr":"172.21.15.57:6807/113976","heartbeat_front_addr":"172.21.15.57:6805/113976","state":["exists","up"]},{"osd":1,"uuid":"e2dc7148-fc1a-48a4-8b06-013164077813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":113977},{"type":"v1","addr":"172.21.15.57:6809","nonce":113977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":113977},{"type":"v1","addr":"172.21.15.57:6811","nonce":113977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":113977},{"type":"v1","addr":"172.21.15.57:6815","nonce":113977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":113977},{"type":"v1","addr":"172.21.15.57:6813","nonce":113977}]},"public_addr":"172.21.15.57:6809/113977","cluster_addr":"172.21.15.57:6811/113977","heartbeat_back_addr":"172.21.15.57:6815/113977","heartbeat_front_addr":"172.21.15.57:6813/113977","state":["exists","up"]},{"osd":2,"uuid":"d30552e4-8c21-4ff9-baf7-07c7ed117bdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6817","nonce":113978},{"type":"v1","addr":"172.21.15.57:6819","nonce":113978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6821","nonce":113978},{"type":"v1","addr":"172.21.15.57:6823","nonce":113978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6829","nonce":113978},{"type":"v1","addr":"172.21.15.57:6831","nonce":113978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6825","nonce":113978},{"type":"v1","addr":"172.21.15.57:6827","nonce":113978}]},"public_addr":"172.21.15.57:6819/113978","cluster_addr":"172.21.15.57:6823/113978","heartbeat_back_addr":"172.21.15.57:6831/113978","heartbeat_front_addr":"172.21.15.57:6827/113978","state":["exists","up"]},{"osd":3,"uuid":"d22728ed-0946-49b6-871d-ab80ec2089c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":113995},{"type":"v1","addr":"172.21.15.57:6818","nonce":113995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":113995},{"type":"v1","addr":"172.21.15.57:6822","nonce":113995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":113995},{"type":"v1","addr":"172.21.15.57:6830","nonce":113995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":113995},{"type":"v1","addr":"172.21.15.57:6826","nonce":113995}]},"public_addr":"172.21.15.57:6818/113995","cluster_addr":"172.21.15.57:6822/113995","heartbeat_back_addr":"172.21.15.57:6830/113995","heartbeat_front_addr":"172.21.15.57:6826/113995","state":["exists","up"]},{"osd":4,"uuid":"a39e2e51-40b1-4c91-b045-f6f030dd6546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":114171},{"type":"v1","addr":"172.21.15.123:6801","nonce":114171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6803","nonce":114171},{"type":"v1","addr":"172.21.15.123:6805","nonce":114171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6811","nonce":114171},{"type":"v1","addr":"172.21.15.123:6813","nonce":114171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6807","nonce":114171},{"type":"v1","addr":"172.21.15.123:6809","nonce":114171}]},"public_addr":"172.21.15.123:6801/114171","cluster_addr":"172.21.15.123:6805/114171","heartbeat_back_addr":"172.21.15.123:6813/114171","heartbeat_front_addr":"172.21.15.123:6809/114171","state":["exists","up"]},{"osd":5,"uuid":"855f67e8-1b47-4fb7-a8c2-2f5df954a8ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":114172},{"type":"v1","addr":"172.21.15.123:6804","nonce":114172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":114172},{"type":"v1","addr":"172.21.15.123:6808","nonce":114172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":114172},{"type":"v1","addr":"172.21.15.123:6815","nonce":114172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":114172},{"type":"v1","addr":"172.21.15.123:6812","nonce":114172}]},"public_addr":"172.21.15.123:6804/114172","cluster_addr":"172.21.15.123:6808/114172","heartbeat_back_addr":"172.21.15.123:6815/114172","heartbeat_front_addr":"172.21.15.123:6812/114172","state":["exists","up"]},{"osd":6,"uuid":"a74edd8e-c135-473e-9394-f214d0405572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":114216},{"type":"v1","addr":"172.21.15.123:6825","nonce":114216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":114216},{"type":"v1","addr":"172.21.15.123:6827","nonce":114216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":114216},{"type":"v1","addr":"172.21.15.123:6831","nonce":114216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":114216},{"type":"v1","addr":"172.21.15.123:6829","nonce":114216}]},"public_addr":"172.21.15.123:6825/114216","cluster_addr":"172.21.15.123:6827/114216","heartbeat_back_addr":"172.21.15.123:6831/114216","heartbeat_front_addr":"172.21.15.123:6829/114216","state":["exists","up"]},{"osd":7,"uuid":"26725e66-b047-4e78-afc6-352a8ea70559","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":114173},{"type":"v1","addr":"172.21.15.123:6817","nonce":114173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":114173},{"type":"v1","addr":"172.21.15.123:6819","nonce":114173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":114173},{"type":"v1","addr":"172.21.15.123:6823","nonce":114173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":114173},{"type":"v1","addr":"172.21.15.123:6821","nonce":114173}]},"public_addr":"172.21.15.123:6817/114173","cluster_addr":"172.21.15.123:6819/114173","heartbeat_back_addr":"172.21.15.123:6823/114173","heartbeat_front_addr":"172.21.15.123:6821/114173","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.786276+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.749336+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.794142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.824374+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.812739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.708934+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.806842+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.766984+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:19:54.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8ad4066e90 msgr2=0x7f8ad4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8ad4066e90 0x7f8ad4069340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8adc006fd0 tx=0x7f8adc008040 comp rx=0 tx=0).stop 2023-06-23T01:19:54.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 msgr2=0x7f8af416ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af416ca20 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8ae400c510 tx=0x7f8ae400c8d0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 shutdown_connections 2023-06-23T01:19:54.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f8ad4066e90 0x7f8ad4069340 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.071 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8af40c8d70 0x7f8af414f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f8af40b18e0 0x7f8af414ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 --2- 172.21.15.57:0/3738950677 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f8af40b1cb0 0x7f8af416ca20 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 >> 172.21.15.57:0/3738950677 conn(0x7f8af4070220 msgr2=0x7f8af40cbfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:54.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 shutdown_connections 2023-06-23T01:19:54.072 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.067+0000 7f8af9731700 1 -- 172.21.15.57:0/3738950677 wait complete. 2023-06-23T01:19:54.082 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-06-23T01:19:50.441526+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:19:50.876930+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2023-06-23T01:19:54.084 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T01:19:54.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.205+0000 7f87d9257700 1 Processor -- start 2023-06-23T01:19:54.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.205+0000 7f87d9257700 1 -- start start 2023-06-23T01:19:54.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.205+0000 7f87d9257700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f87d4150480 0x7f87d41565e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.205+0000 7f87d9257700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87d40cc5c0 0x7f87d40cca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.205+0000 7f87d9257700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d406da50 con 0x7f87d4150850 2023-06-23T01:19:54.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.205+0000 7f87d9257700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d40ccf80 con 0x7f87d40cc5c0 2023-06-23T01:19:54.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.205+0000 7f87d9257700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d415a430 con 0x7f87d4150480 2023-06-23T01:19:54.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d359e700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87d40cc5c0 0x7f87d40cca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d259c700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f87d4150850 0x7f87d4159be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47210/0 (socket says 172.21.15.57:47210) 2023-06-23T01:19:54.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d359e700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f87d40cc5c0 0x7f87d40cca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42780/0 (socket says 172.21.15.57:42780) 2023-06-23T01:19:54.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d259c700 1 -- 172.21.15.57:0/550365326 learned_addr learned my addr 172.21.15.57:0/550365326 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:54.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d2d9d700 1 --2- 172.21.15.57:0/550365326 >> v2:172.21.15.123:3300/0 conn(0x7f87d4150480 0x7f87d41565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d359e700 1 -- 172.21.15.57:0/550365326 >> v2:172.21.15.123:3300/0 conn(0x7f87d4150480 msgr2=0x7f87d41565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:54.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d359e700 1 --2- 172.21.15.57:0/550365326 >> v2:172.21.15.123:3300/0 conn(0x7f87d4150480 0x7f87d41565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d359e700 1 -- 172.21.15.57:0/550365326 >> v1:172.21.15.57:6789/0 conn(0x7f87d4150850 legacy=0x7f87d4159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d359e700 1 -- 172.21.15.57:0/550365326 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d415a710 con 0x7f87d40cc5c0 2023-06-23T01:19:54.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d1d9b700 1 -- 172.21.15.57:0/550365326 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802315346 0 0) 0x7f87d406da50 con 0x7f87d4150850 2023-06-23T01:19:54.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d359e700 1 --2- 172.21.15.57:0/550365326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d40cc5c0 0x7f87d40cca40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009e60 tx=0x7f87cc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8e5d6982f432e7f server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d1d9b700 1 -- 172.21.15.57:0/550365326 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc00e070 con 0x7f87d40cc5c0 2023-06-23T01:19:54.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d1d9b700 1 -- 172.21.15.57:0/550365326 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87cc00bbf0 con 0x7f87d40cc5c0 2023-06-23T01:19:54.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.206+0000 7f87d1d9b700 1 -- 172.21.15.57:0/550365326 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87cc012430 con 0x7f87d40cc5c0 2023-06-23T01:19:54.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.207+0000 7f87d9257700 1 -- 172.21.15.57:0/550365326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d40cc5c0 msgr2=0x7f87d40cca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.207+0000 7f87d9257700 1 --2- 172.21.15.57:0/550365326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d40cc5c0 0x7f87d40cca40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f87cc009e60 tx=0x7f87cc009780 comp rx=0 tx=0).stop 2023-06-23T01:19:54.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.207+0000 7f87d9257700 1 -- 172.21.15.57:0/550365326 shutdown_connections 2023-06-23T01:19:54.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.207+0000 7f87d9257700 1 --2- 172.21.15.57:0/550365326 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d40cc5c0 0x7f87d40cca40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.207+0000 7f87d9257700 1 --2- 172.21.15.57:0/550365326 >> v2:172.21.15.123:3300/0 conn(0x7f87d4150480 0x7f87d41565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.207+0000 7f87d9257700 1 -- 172.21.15.57:0/550365326 >> 172.21.15.57:0/550365326 conn(0x7f87d4070220 msgr2=0x7f87d4070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:54.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.207+0000 7f87d9257700 1 -- 172.21.15.57:0/550365326 shutdown_connections 2023-06-23T01:19:54.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.207+0000 7f87d9257700 1 -- 172.21.15.57:0/550365326 wait complete. 2023-06-23T01:19:54.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.208+0000 7f87d9257700 1 Processor -- start 2023-06-23T01:19:54.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.208+0000 7f87d9257700 1 -- start start 2023-06-23T01:19:54.217 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d9257700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87d40cc5c0 0x7f87d414e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d9257700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d4150480 0x7f87d415ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d9257700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87d4150850 0x7f87d40cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d9257700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d414e610 con 0x7f87d4150480 2023-06-23T01:19:54.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d9257700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d40cb3b0 con 0x7f87d4150850 2023-06-23T01:19:54.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d9257700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d40ca7a0 con 0x7f87d40cc5c0 2023-06-23T01:19:54.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d4150480 0x7f87d415ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.218 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d4150480 0x7f87d415ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42794/0 (socket says 172.21.15.57:42794) 2023-06-23T01:19:54.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 -- 172.21.15.57:0/1367156223 learned_addr learned my addr 172.21.15.57:0/1367156223 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:54.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d359e700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87d4150850 0x7f87d40cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d2d9d700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87d40cc5c0 0x7f87d414e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 -- 172.21.15.57:0/1367156223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87d40cc5c0 msgr2=0x7f87d414e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87d40cc5c0 0x7f87d414e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 -- 172.21.15.57:0/1367156223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87d4150850 msgr2=0x7f87d40cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87d4150850 0x7f87d40cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.219 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 -- 172.21.15.57:0/1367156223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87cc009a20 con 0x7f87d4150480 2023-06-23T01:19:54.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.209+0000 7f87d259c700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d4150480 0x7f87d415ad00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f87c800c7e0 tx=0x7f87c8007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.210+0000 7f87baffd700 1 -- 172.21.15.57:0/1367156223 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8005b30 con 0x7f87d4150480 2023-06-23T01:19:54.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.210+0000 7f87baffd700 1 -- 172.21.15.57:0/1367156223 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c8005c90 con 0x7f87d4150480 2023-06-23T01:19:54.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.210+0000 7f87baffd700 1 -- 172.21.15.57:0/1367156223 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87c8007300 con 0x7f87d4150480 2023-06-23T01:19:54.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.210+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d40caa80 con 0x7f87d4150480 2023-06-23T01:19:54.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.210+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d40cb040 con 0x7f87d4150480 2023-06-23T01:19:54.220 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.210+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d40cb520 con 0x7f87d4150480 2023-06-23T01:19:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.211+0000 7f87baffd700 1 -- 172.21.15.57:0/1367156223 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f87c80076b0 con 0x7f87d4150480 2023-06-23T01:19:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.211+0000 7f87baffd700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f87c0066e90 0x7f87c0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.211+0000 7f87baffd700 1 -- 172.21.15.57:0/1367156223 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f87c80837e0 con 0x7f87d4150480 2023-06-23T01:19:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.211+0000 7f87d2d9d700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f87c0066e90 0x7f87c0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.212+0000 7f87d2d9d700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f87c0066e90 0x7f87c0069340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f87c4006fd0 tx=0x7f87c4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.221 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.216+0000 7f87baffd700 1 -- 172.21.15.57:0/1367156223 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f87c8050c60 con 0x7f87d4150480 2023-06-23T01:19:54.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.409+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f87d4064b50 con 0x7f87d4150480 2023-06-23T01:19:54.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.409+0000 7f87baffd700 1 -- 172.21.15.57:0/1367156223 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7f87c8017020 con 0x7f87d4150480 2023-06-23T01:19:54.413 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 8 2023-06-23T01:19:54.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f87c0066e90 msgr2=0x7f87c0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f87c0066e90 0x7f87c0069340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f87c4006fd0 tx=0x7f87c4008040 comp rx=0 tx=0).stop 2023-06-23T01:19:54.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d4150480 msgr2=0x7f87d415ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d4150480 0x7f87d415ad00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f87c800c7e0 tx=0x7f87c8007df0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 shutdown_connections 2023-06-23T01:19:54.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f87c0066e90 0x7f87c0069340 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f87d4150850 0x7f87d40cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f87d40cc5c0 0x7f87d414e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 --2- 172.21.15.57:0/1367156223 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f87d4150480 0x7f87d415ad00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 >> 172.21.15.57:0/1367156223 conn(0x7f87d4070220 msgr2=0x7f87d4157ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:54.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 shutdown_connections 2023-06-23T01:19:54.417 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.411+0000 7f87d9257700 1 -- 172.21.15.57:0/1367156223 wait complete. 2023-06-23T01:19:54.427 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:19:54.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.548+0000 7f4a6084a700 1 Processor -- start 2023-06-23T01:19:54.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.548+0000 7f4a6084a700 1 -- start start 2023-06-23T01:19:54.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a6084a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c140940 0x7f4a5c0ac9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a6084a700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4a5c0b1ed0 0x7f4a5c0b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a6084a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a5c0b2c70 con 0x7f4a5c0acf00 2023-06-23T01:19:54.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a6084a700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a5c0b2870 con 0x7f4a5c0b1ed0 2023-06-23T01:19:54.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a6084a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c15b6f0 con 0x7f4a5c140940 2023-06-23T01:19:54.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5ad9d700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4a5c0b1ed0 0x7f4a5c0b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5ad9d700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4a5c0b1ed0 0x7f4a5c0b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42810/0 (socket says 172.21.15.57:42810) 2023-06-23T01:19:54.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5ad9d700 1 -- 172.21.15.57:0/373249813 learned_addr learned my addr 172.21.15.57:0/373249813 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:54.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5a59c700 1 --2- 172.21.15.57:0/373249813 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c140940 0x7f4a5c0ac9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.555 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5ad9d700 1 -- 172.21.15.57:0/373249813 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c140940 msgr2=0x7f4a5c0ac9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:54.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5ad9d700 1 --2- 172.21.15.57:0/373249813 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c140940 0x7f4a5c0ac9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5ad9d700 1 -- 172.21.15.57:0/373249813 >> v1:172.21.15.123:6789/0 conn(0x7f4a5c0acf00 legacy=0x7f4a5c15ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5ad9d700 1 -- 172.21.15.57:0/373249813 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a5c15b9d0 con 0x7f4a5c0b1ed0 2023-06-23T01:19:54.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.549+0000 7f4a5a59c700 1 --2- 172.21.15.57:0/373249813 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c140940 0x7f4a5c0ac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:19:54.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a5ad9d700 1 --2- 172.21.15.57:0/373249813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c0b1ed0 0x7f4a5c0b2330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4a54009f20 tx=0x7f4a54009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7712e57616e3787e server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a5959a700 1 -- 172.21.15.57:0/373249813 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5400e070 con 0x7f4a5c0b1ed0 2023-06-23T01:19:54.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a5959a700 1 -- 172.21.15.57:0/373249813 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a5400ba80 con 0x7f4a5c0b1ed0 2023-06-23T01:19:54.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a5959a700 1 -- 172.21.15.57:0/373249813 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a540092e0 con 0x7f4a5c0b1ed0 2023-06-23T01:19:54.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a6084a700 1 -- 172.21.15.57:0/373249813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c0b1ed0 msgr2=0x7f4a5c0b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a6084a700 1 --2- 172.21.15.57:0/373249813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c0b1ed0 0x7f4a5c0b2330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4a54009f20 tx=0x7f4a54009840 comp rx=0 tx=0).stop 2023-06-23T01:19:54.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a6084a700 1 -- 172.21.15.57:0/373249813 shutdown_connections 2023-06-23T01:19:54.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a6084a700 1 --2- 172.21.15.57:0/373249813 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c0b1ed0 0x7f4a5c0b2330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a6084a700 1 --2- 172.21.15.57:0/373249813 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c140940 0x7f4a5c0ac9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a6084a700 1 -- 172.21.15.57:0/373249813 >> 172.21.15.57:0/373249813 conn(0x7f4a5c06eba0 msgr2=0x7f4a5c06efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:54.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a6084a700 1 -- 172.21.15.57:0/373249813 shutdown_connections 2023-06-23T01:19:54.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.550+0000 7f4a6084a700 1 -- 172.21.15.57:0/373249813 wait complete. 2023-06-23T01:19:54.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a6084a700 1 Processor -- start 2023-06-23T01:19:54.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a6084a700 1 -- start start 2023-06-23T01:19:54.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a6084a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c0acf00 0x7f4a5c16afb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a6084a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a5c0b1ed0 0x7f4a5c1566b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a6084a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c140940 0x7f4a5c069ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a6084a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c0aac20 con 0x7f4a5c140940 2023-06-23T01:19:54.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a6084a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c16b4f0 con 0x7f4a5c0b1ed0 2023-06-23T01:19:54.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a6084a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a5c0aaec0 con 0x7f4a5c0acf00 2023-06-23T01:19:54.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a5ad9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c140940 0x7f4a5c069ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a5ad9d700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c140940 0x7f4a5c069ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42816/0 (socket says 172.21.15.57:42816) 2023-06-23T01:19:54.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.552+0000 7f4a5ad9d700 1 -- 172.21.15.57:0/4175320114 learned_addr learned my addr 172.21.15.57:0/4175320114 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:54.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a5a59c700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c0acf00 0x7f4a5c16afb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a59d9b700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a5c0b1ed0 0x7f4a5c1566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a5ad9d700 1 -- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c0acf00 msgr2=0x7f4a5c16afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a5ad9d700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c0acf00 0x7f4a5c16afb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a5ad9d700 1 -- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a5c0b1ed0 msgr2=0x7f4a5c1566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a5ad9d700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a5c0b1ed0 0x7f4a5c1566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a5ad9d700 1 -- 172.21.15.57:0/4175320114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a54009ae0 con 0x7f4a5c140940 2023-06-23T01:19:54.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a59d9b700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a5c0b1ed0 0x7f4a5c1566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:19:54.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a5a59c700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c0acf00 0x7f4a5c16afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:19:54.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a5ad9d700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c140940 0x7f4a5c069ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400b6c0 tx=0x7f4a54018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a4a7fc700 1 -- 172.21.15.57:0/4175320114 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54025460 con 0x7f4a5c140940 2023-06-23T01:19:54.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a4a7fc700 1 -- 172.21.15.57:0/4175320114 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a540255c0 con 0x7f4a5c140940 2023-06-23T01:19:54.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a5c0ab1a0 con 0x7f4a5c140940 2023-06-23T01:19:54.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a4a7fc700 1 -- 172.21.15.57:0/4175320114 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a54025980 con 0x7f4a5c140940 2023-06-23T01:19:54.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.553+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a5c0a7720 con 0x7f4a5c140940 2023-06-23T01:19:54.563 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.554+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a5c064b50 con 0x7f4a5c140940 2023-06-23T01:19:54.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.554+0000 7f4a4a7fc700 1 -- 172.21.15.57:0/4175320114 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f4a54003040 con 0x7f4a5c140940 2023-06-23T01:19:54.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.555+0000 7f4a4a7fc700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4a3c066e90 0x7f4a3c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.555+0000 7f4a4a7fc700 1 -- 172.21.15.57:0/4175320114 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f4a540860a0 con 0x7f4a5c140940 2023-06-23T01:19:54.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.555+0000 7f4a5a59c700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4a3c066e90 0x7f4a3c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.564 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.555+0000 7f4a5a59c700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4a3c066e90 0x7f4a3c069340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4a50008b20 tx=0x7f4a50008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.565 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.559+0000 7f4a4a7fc700 1 -- 172.21.15.57:0/4175320114 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f4a540535c0 con 0x7f4a5c140940 2023-06-23T01:19:54.756 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.752+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a5c0b16d0 con 0x7f4a5c140940 2023-06-23T01:19:54.757 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.753+0000 7f4a4a7fc700 1 -- 172.21.15.57:0/4175320114 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a54056be0 con 0x7f4a5c140940 2023-06-23T01:19:54.757 INFO:teuthology.orchestra.run.smithi057.stdout:pg_num: 1 2023-06-23T01:19:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4a3c066e90 msgr2=0x7f4a3c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4a3c066e90 0x7f4a3c069340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4a50008b20 tx=0x7f4a50008040 comp rx=0 tx=0).stop 2023-06-23T01:19:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c140940 msgr2=0x7f4a5c069ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c140940 0x7f4a5c069ac0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4a5400b6c0 tx=0x7f4a54018000 comp rx=0 tx=0).stop 2023-06-23T01:19:54.759 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 shutdown_connections 2023-06-23T01:19:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4a3c066e90 0x7f4a3c069340 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4a5c140940 0x7f4a5c069ac0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4a5c0acf00 0x7f4a5c16afb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 --2- 172.21.15.57:0/4175320114 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a5c0b1ed0 0x7f4a5c1566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 >> 172.21.15.57:0/4175320114 conn(0x7f4a5c06eba0 msgr2=0x7f4a5c063020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 shutdown_connections 2023-06-23T01:19:54.760 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.755+0000 7f4a6084a700 1 -- 172.21.15.57:0/4175320114 wait complete. 2023-06-23T01:19:54.770 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-23T01:19:54.892 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.888+0000 7fd239ed3700 1 Processor -- start 2023-06-23T01:19:54.893 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.889+0000 7fd239ed3700 1 -- start start 2023-06-23T01:19:54.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.889+0000 7fd239ed3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd2340cf000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.889+0000 7fd239ed3700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd2340cb300 0x7fd2340ccc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.889+0000 7fd239ed3700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2340cf630 con 0x7fd2340af300 2023-06-23T01:19:54.894 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.889+0000 7fd239ed3700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2340cf910 con 0x7fd2340cb300 2023-06-23T01:19:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.889+0000 7fd239ed3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2340cfbf0 con 0x7fd2340af6d0 2023-06-23T01:19:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd232ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd2340cf000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd233fff700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fd2340cb300 0x7fd2340ccc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd232ffd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd2340cf000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.123:3301/0 says I am v2:172.21.15.57:43242/0 (socket says 172.21.15.57:43242) 2023-06-23T01:19:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd232ffd700 1 -- 172.21.15.57:0/1076411340 learned_addr learned my addr 172.21.15.57:0/1076411340 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd2337fe700 1 --1- 172.21.15.57:0/1076411340 >> v1:172.21.15.123:6789/0 conn(0x7fd2340af300 0x7fd2340d06d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.123:6789/0 says I am v1:172.21.15.57:39674/0 (socket says 172.21.15.57:39674) 2023-06-23T01:19:54.895 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd233fff700 1 -- 172.21.15.57:0/1076411340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 msgr2=0x7fd2340cf000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd233fff700 1 --2- 172.21.15.57:0/1076411340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd2340cf000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd233fff700 1 -- 172.21.15.57:0/1076411340 >> v1:172.21.15.123:6789/0 conn(0x7fd2340af300 legacy=0x7fd2340d06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd233fff700 1 -- 172.21.15.57:0/1076411340 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2340cfed0 con 0x7fd2340cb300 2023-06-23T01:19:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd232ffd700 1 --2- 172.21.15.57:0/1076411340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd2340cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd2327fc700 1 -- 172.21.15.57:0/1076411340 <== mon.1 v1:172.21.15.123:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408795304 0 0) 0x7fd2340cf630 con 0x7fd2340af300 2023-06-23T01:19:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.890+0000 7fd233fff700 1 --2- 172.21.15.57:0/1076411340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340cb300 0x7fd2340ccc10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009e60 tx=0x7fd22c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa4aebdf105e0754 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.896 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd2327fc700 1 -- 172.21.15.57:0/1076411340 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c00e070 con 0x7fd2340cb300 2023-06-23T01:19:54.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd2327fc700 1 -- 172.21.15.57:0/1076411340 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c00bb60 con 0x7fd2340cb300 2023-06-23T01:19:54.897 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd2327fc700 1 -- 172.21.15.57:0/1076411340 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c012430 con 0x7fd2340cb300 2023-06-23T01:19:54.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd239ed3700 1 -- 172.21.15.57:0/1076411340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340cb300 msgr2=0x7fd2340ccc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd239ed3700 1 --2- 172.21.15.57:0/1076411340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340cb300 0x7fd2340ccc10 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd22c009e60 tx=0x7fd22c009780 comp rx=0 tx=0).stop 2023-06-23T01:19:54.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd239ed3700 1 -- 172.21.15.57:0/1076411340 shutdown_connections 2023-06-23T01:19:54.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd239ed3700 1 --2- 172.21.15.57:0/1076411340 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340cb300 0x7fd2340ccc10 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.899 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd239ed3700 1 --2- 172.21.15.57:0/1076411340 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd2340cf000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd239ed3700 1 -- 172.21.15.57:0/1076411340 >> 172.21.15.57:0/1076411340 conn(0x7fd234067340 msgr2=0x7fd234067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd239ed3700 1 -- 172.21.15.57:0/1076411340 shutdown_connections 2023-06-23T01:19:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.891+0000 7fd239ed3700 1 -- 172.21.15.57:0/1076411340 wait complete. 2023-06-23T01:19:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.892+0000 7fd239ed3700 1 Processor -- start 2023-06-23T01:19:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd239ed3700 1 -- start start 2023-06-23T01:19:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd239ed3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340af300 0x7fd234157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.900 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd239ed3700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd23416bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd239ed3700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2340cb300 0x7fd23416c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd239ed3700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd23416c640 con 0x7fd2340cb300 2023-06-23T01:19:54.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd239ed3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd234156460 con 0x7fd2340af300 2023-06-23T01:19:54.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd239ed3700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd234156740 con 0x7fd2340af6d0 2023-06-23T01:19:54.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd233fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2340cb300 0x7fd23416c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd233fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2340cb300 0x7fd23416c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42822/0 (socket says 172.21.15.57:42822) 2023-06-23T01:19:54.901 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd233fff700 1 -- 172.21.15.57:0/2379994910 learned_addr learned my addr 172.21.15.57:0/2379994910 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:54.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.893+0000 7fd232ffd700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd23416bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd2337fe700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340af300 0x7fd234157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd233fff700 1 -- 172.21.15.57:0/2379994910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 msgr2=0x7fd23416bbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd233fff700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd23416bbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd233fff700 1 -- 172.21.15.57:0/2379994910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340af300 msgr2=0x7fd234157310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:54.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd233fff700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340af300 0x7fd234157310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:54.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd233fff700 1 -- 172.21.15.57:0/2379994910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd21c0094f0 con 0x7fd2340cb300 2023-06-23T01:19:54.902 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd233fff700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2340cb300 0x7fd23416c100 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd2340cdd90 tx=0x7fd22c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd22b7fe700 1 -- 172.21.15.57:0/2379994910 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c012ea0 con 0x7fd2340cb300 2023-06-23T01:19:54.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd22c009a20 con 0x7fd2340cb300 2023-06-23T01:19:54.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd22b7fe700 1 -- 172.21.15.57:0/2379994910 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd22c027470 con 0x7fd2340cb300 2023-06-23T01:19:54.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd22b7fe700 1 -- 172.21.15.57:0/2379994910 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd22c027830 con 0x7fd2340cb300 2023-06-23T01:19:54.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.894+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd234152590 con 0x7fd2340cb300 2023-06-23T01:19:54.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.895+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd234064b50 con 0x7fd2340cb300 2023-06-23T01:19:54.903 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.895+0000 7fd22b7fe700 1 -- 172.21.15.57:0/2379994910 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fd22c024070 con 0x7fd2340cb300 2023-06-23T01:19:54.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.896+0000 7fd22b7fe700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fd214066e90 0x7fd214069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:54.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.896+0000 7fd22b7fe700 1 -- 172.21.15.57:0/2379994910 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fd22c086980 con 0x7fd2340cb300 2023-06-23T01:19:54.904 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.896+0000 7fd2337fe700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fd214066e90 0x7fd214069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:54.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.896+0000 7fd2337fe700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fd214066e90 0x7fd214069340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd224007180 tx=0x7fd224007ef0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:54.905 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:54.900+0000 7fd22b7fe700 1 -- 172.21.15.57:0/2379994910 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fd22c053ea0 con 0x7fd2340cb300 2023-06-23T01:19:55.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.098+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd2340d01b0 con 0x7fd2340cb300 2023-06-23T01:19:55.102 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.099+0000 7fd22b7fe700 1 -- 172.21.15.57:0/2379994910 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+126 (secure 0 0 0) 0x7fd22c02c090 con 0x7fd2340cb300 2023-06-23T01:19:55.103 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:55.103 INFO:teuthology.orchestra.run.smithi057.stdout:[ 2023-06-23T01:19:55.103 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-06-23T01:19:55.103 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 1, 2023-06-23T01:19:55.103 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": "rbd" 2023-06-23T01:19:55.103 INFO:teuthology.orchestra.run.smithi057.stdout: }, 2023-06-23T01:19:55.104 INFO:teuthology.orchestra.run.smithi057.stdout: { 2023-06-23T01:19:55.104 INFO:teuthology.orchestra.run.smithi057.stdout: "poolnum": 2, 2023-06-23T01:19:55.105 INFO:teuthology.orchestra.run.smithi057.stdout: "poolname": ".mgr" 2023-06-23T01:19:55.105 INFO:teuthology.orchestra.run.smithi057.stdout: } 2023-06-23T01:19:55.105 INFO:teuthology.orchestra.run.smithi057.stdout:] 2023-06-23T01:19:55.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fd214066e90 msgr2=0x7fd214069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:55.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fd214066e90 0x7fd214069340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd224007180 tx=0x7fd224007ef0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2340cb300 msgr2=0x7fd23416c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:55.106 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2340cb300 0x7fd23416c100 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd2340cdd90 tx=0x7fd22c019000 comp rx=0 tx=0).stop 2023-06-23T01:19:55.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 shutdown_connections 2023-06-23T01:19:55.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fd214066e90 0x7fd214069340 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fd2340cb300 0x7fd23416c100 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd2340af300 0x7fd234157310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 --2- 172.21.15.57:0/2379994910 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fd2340af6d0 0x7fd23416bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.107 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 >> 172.21.15.57:0/2379994910 conn(0x7fd234067340 msgr2=0x7fd234151ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:55.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 shutdown_connections 2023-06-23T01:19:55.108 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.101+0000 7fd239ed3700 1 -- 172.21.15.57:0/2379994910 wait complete. 2023-06-23T01:19:55.116 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-23T01:19:55.117 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-23T01:19:55.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.240+0000 7fc5fa056700 1 Processor -- start 2023-06-23T01:19:55.244 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.240+0000 7fc5fa056700 1 -- start start 2023-06-23T01:19:55.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5fa056700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc5f40b22e0 0x7fc5f40b5a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:55.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5fa056700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f415afb0 con 0x7fc5f40b5470 2023-06-23T01:19:55.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5fa056700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5f415bb30 con 0x7fc5f40b3830 2023-06-23T01:19:55.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5fa056700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5f40b5840 con 0x7fc5f40b22e0 2023-06-23T01:19:55.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5ebfff700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7fc5f40b22e0 0x7fc5f40b5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:55.246 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5f37fe700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fc5f40b5470 0x7fc5f40b1bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:47216/0 (socket says 172.21.15.57:47216) 2023-06-23T01:19:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5f37fe700 1 -- 172.21.15.57:0/2943336850 learned_addr learned my addr 172.21.15.57:0/2943336850 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5ebfff700 1 -- 172.21.15.57:0/2943336850 >> v1:172.21.15.123:6789/0 conn(0x7fc5f40b3830 legacy=0x7fc5f415bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5ebfff700 1 -- 172.21.15.57:0/2943336850 >> v1:172.21.15.57:6789/0 conn(0x7fc5f40b5470 legacy=0x7fc5f40b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5ebfff700 1 -- 172.21.15.57:0/2943336850 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5f415b200 con 0x7fc5f40b22e0 2023-06-23T01:19:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5ebfff700 1 --2- 172.21.15.57:0/2943336850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b22e0 0x7fc5f40b5a80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0098e0 tx=0x7fc5dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7b5c7551faa15f server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5f2ffd700 1 -- 172.21.15.57:0/2943336850 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc009e80 con 0x7fc5f40b22e0 2023-06-23T01:19:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5f2ffd700 1 -- 172.21.15.57:0/2943336850 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5dc00b860 con 0x7fc5f40b22e0 2023-06-23T01:19:55.247 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.242+0000 7fc5f2ffd700 1 -- 172.21.15.57:0/2943336850 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc00bbe0 con 0x7fc5f40b22e0 2023-06-23T01:19:55.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.243+0000 7fc5fa056700 1 -- 172.21.15.57:0/2943336850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b22e0 msgr2=0x7fc5f40b5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:55.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.243+0000 7fc5fa056700 1 --2- 172.21.15.57:0/2943336850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b22e0 0x7fc5f40b5a80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc0098e0 tx=0x7fc5dc0091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.243+0000 7fc5fa056700 1 -- 172.21.15.57:0/2943336850 shutdown_connections 2023-06-23T01:19:55.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.243+0000 7fc5fa056700 1 --2- 172.21.15.57:0/2943336850 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b22e0 0x7fc5f40b5a80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.243+0000 7fc5fa056700 1 -- 172.21.15.57:0/2943336850 >> 172.21.15.57:0/2943336850 conn(0x7fc5f4067340 msgr2=0x7fc5f4067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:55.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.243+0000 7fc5fa056700 1 -- 172.21.15.57:0/2943336850 shutdown_connections 2023-06-23T01:19:55.250 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.243+0000 7fc5fa056700 1 -- 172.21.15.57:0/2943336850 wait complete. 2023-06-23T01:19:55.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.244+0000 7fc5fa056700 1 Processor -- start 2023-06-23T01:19:55.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.244+0000 7fc5fa056700 1 -- start start 2023-06-23T01:19:55.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5fa056700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f40b22e0 0x7fc5f416ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:55.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5fa056700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5f40b3830 0x7fc5f416c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:55.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5fa056700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b5470 0x7fc5f40c9d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:55.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5fa056700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f4169920 con 0x7fc5f40b5470 2023-06-23T01:19:55.251 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5fa056700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f416b070 con 0x7fc5f40b22e0 2023-06-23T01:19:55.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5fa056700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5f40c52f0 con 0x7fc5f40b3830 2023-06-23T01:19:55.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b5470 0x7fc5f40c9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:55.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f3fff700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b5470 0x7fc5f40c9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42846/0 (socket says 172.21.15.57:42846) 2023-06-23T01:19:55.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f3fff700 1 -- 172.21.15.57:0/1301263899 learned_addr learned my addr 172.21.15.57:0/1301263899 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:55.252 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f37fe700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f40b22e0 0x7fc5f416ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:55.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5ebfff700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5f40b3830 0x7fc5f416c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:55.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f3fff700 1 -- 172.21.15.57:0/1301263899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5f40b3830 msgr2=0x7fc5f416c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:55.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f3fff700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5f40b3830 0x7fc5f416c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f3fff700 1 -- 172.21.15.57:0/1301263899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f40b22e0 msgr2=0x7fc5f416ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:55.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f3fff700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f40b22e0 0x7fc5f416ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:55.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.245+0000 7fc5f3fff700 1 -- 172.21.15.57:0/1301263899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc009420 con 0x7fc5f40b5470 2023-06-23T01:19:55.253 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.246+0000 7fc5f3fff700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b5470 0x7fc5f40c9d20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00c8e0 tx=0x7fc5ec001710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:55.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.246+0000 7fc5eb7fe700 1 -- 172.21.15.57:0/1301263899 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec012630 con 0x7fc5f40b5470 2023-06-23T01:19:55.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.246+0000 7fc5eb7fe700 1 -- 172.21.15.57:0/1301263899 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec012790 con 0x7fc5f40b5470 2023-06-23T01:19:55.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.246+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5f40c55d0 con 0x7fc5f40b5470 2023-06-23T01:19:55.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.246+0000 7fc5eb7fe700 1 -- 172.21.15.57:0/1301263899 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec012b50 con 0x7fc5f40b5470 2023-06-23T01:19:55.254 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.246+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5f40c5b90 con 0x7fc5f40b5470 2023-06-23T01:19:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.247+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5f4169a60 con 0x7fc5f40b5470 2023-06-23T01:19:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.247+0000 7fc5eb7fe700 1 -- 172.21.15.57:0/1301263899 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fc5ec01a3a0 con 0x7fc5f40b5470 2023-06-23T01:19:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.247+0000 7fc5eb7fe700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fc5d4066e90 0x7fc5d4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.247+0000 7fc5eb7fe700 1 -- 172.21.15.57:0/1301263899 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc5ec0843c0 con 0x7fc5f40b5470 2023-06-23T01:19:55.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.247+0000 7fc5f37fe700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fc5d4066e90 0x7fc5d4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:55.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.248+0000 7fc5f37fe700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fc5d4066e90 0x7fc5d4069340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4001950 tx=0x7fc5e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:55.256 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.252+0000 7fc5eb7fe700 1 -- 172.21.15.57:0/1301263899 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fc5ec055370 con 0x7fc5f40b5470 2023-06-23T01:19:55.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:55.447+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc5f4064b50 con 0x7fc5f40b5470 2023-06-23T01:19:56.255 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.251+0000 7fc5eb7fe700 1 -- 172.21.15.57:0/1301263899 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc5ec054f00 con 0x7fc5f40b5470 2023-06-23T01:19:56.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fc5d4066e90 msgr2=0x7fc5d4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:56.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fc5d4066e90 0x7fc5d4069340 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc5e4001950 tx=0x7fc5e4009040 comp rx=0 tx=0).stop 2023-06-23T01:19:56.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b5470 msgr2=0x7fc5f40c9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:56.258 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b5470 0x7fc5f40c9d20 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec00c8e0 tx=0x7fc5ec001710 comp rx=0 tx=0).stop 2023-06-23T01:19:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 shutdown_connections 2023-06-23T01:19:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fc5d4066e90 0x7fc5d4069340 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fc5f40b5470 0x7fc5f40c9d20 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc5f40b22e0 0x7fc5f416ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 --2- 172.21.15.57:0/1301263899 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fc5f40b3830 0x7fc5f416c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 >> 172.21.15.57:0/1301263899 conn(0x7fc5f4067340 msgr2=0x7fc5f40afa50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:56.259 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.254+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 shutdown_connections 2023-06-23T01:19:56.260 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.255+0000 7fc5fa056700 1 -- 172.21.15.57:0/1301263899 wait complete. 2023-06-23T01:19:56.264 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_metadata' created 2023-06-23T01:19:56.277 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-23T01:19:56.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.409+0000 7f1563640700 1 Processor -- start 2023-06-23T01:19:56.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.409+0000 7f1563640700 1 -- start start 2023-06-23T01:19:56.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.410+0000 7f1563640700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f155c153810 0x7f155c0c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:56.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.410+0000 7f1563640700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f155c159890 0x7f155c0cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.410+0000 7f1563640700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f155c0c9ec0 con 0x7f155c0c72a0 2023-06-23T01:19:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.410+0000 7f1563640700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c0c89d0 con 0x7f155c159890 2023-06-23T01:19:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.410+0000 7f1563640700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f155c159db0 con 0x7f155c153810 2023-06-23T01:19:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.410+0000 7f1560bdb700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f155c159890 0x7f155c0cc590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.410+0000 7f1560bdb700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f155c159890 0x7f155c0cc590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42854/0 (socket says 172.21.15.57:42854) 2023-06-23T01:19:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.410+0000 7f1560bdb700 1 -- 172.21.15.57:0/1507353948 learned_addr learned my addr 172.21.15.57:0/1507353948 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:56.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f15613dc700 1 --2- 172.21.15.57:0/1507353948 >> v2:172.21.15.123:3300/0 conn(0x7f155c153810 0x7f155c0c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f1560bdb700 1 -- 172.21.15.57:0/1507353948 >> v2:172.21.15.123:3300/0 conn(0x7f155c153810 msgr2=0x7f155c0c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f1560bdb700 1 --2- 172.21.15.57:0/1507353948 >> v2:172.21.15.123:3300/0 conn(0x7f155c153810 0x7f155c0c6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f1560bdb700 1 -- 172.21.15.57:0/1507353948 >> v1:172.21.15.123:6789/0 conn(0x7f155c0c72a0 legacy=0x7f155c15ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f1560bdb700 1 -- 172.21.15.57:0/1507353948 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f155c15a090 con 0x7f155c159890 2023-06-23T01:19:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f1560bdb700 1 --2- 172.21.15.57:0/1507353948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c159890 0x7f155c0cc590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f155c06d180 tx=0x7f154c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=355b9f484efede7d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f1553fff700 1 -- 172.21.15.57:0/1507353948 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c00b820 con 0x7f155c159890 2023-06-23T01:19:56.416 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f1553fff700 1 -- 172.21.15.57:0/1507353948 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f154c00b980 con 0x7f155c159890 2023-06-23T01:19:56.418 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.411+0000 7f1553fff700 1 -- 172.21.15.57:0/1507353948 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f154c00bd20 con 0x7f155c159890 2023-06-23T01:19:56.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.412+0000 7f1563640700 1 -- 172.21.15.57:0/1507353948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c159890 msgr2=0x7f155c0cc590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:56.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.412+0000 7f1563640700 1 --2- 172.21.15.57:0/1507353948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c159890 0x7f155c0cc590 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f155c06d180 tx=0x7f154c009250 comp rx=0 tx=0).stop 2023-06-23T01:19:56.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.412+0000 7f1563640700 1 -- 172.21.15.57:0/1507353948 shutdown_connections 2023-06-23T01:19:56.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.412+0000 7f1563640700 1 --2- 172.21.15.57:0/1507353948 >> v2:172.21.15.123:3300/0 conn(0x7f155c153810 0x7f155c0c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.412+0000 7f1563640700 1 --2- 172.21.15.57:0/1507353948 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c159890 0x7f155c0cc590 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.419 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.412+0000 7f1563640700 1 -- 172.21.15.57:0/1507353948 >> 172.21.15.57:0/1507353948 conn(0x7f155c070220 msgr2=0x7f155c070620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:56.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.412+0000 7f1563640700 1 -- 172.21.15.57:0/1507353948 shutdown_connections 2023-06-23T01:19:56.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.412+0000 7f1563640700 1 -- 172.21.15.57:0/1507353948 wait complete. 2023-06-23T01:19:56.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.413+0000 7f1563640700 1 Processor -- start 2023-06-23T01:19:56.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.413+0000 7f1563640700 1 -- start start 2023-06-23T01:19:56.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1563640700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c0c72a0 0x7f155c168dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:56.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1563640700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f155c153810 0x7f155c0c21e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:56.420 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1563640700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f155c159890 0x7f155c0c2720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:56.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1563640700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0c6510 con 0x7f155c0c72a0 2023-06-23T01:19:56.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1563640700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0a9890 con 0x7f155c153810 2023-06-23T01:19:56.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1563640700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f155c0c5890 con 0x7f155c159890 2023-06-23T01:19:56.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c0c72a0 0x7f155c168dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:56.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c0c72a0 0x7f155c168dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:42866/0 (socket says 172.21.15.57:42866) 2023-06-23T01:19:56.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 -- 172.21.15.57:0/4213407699 learned_addr learned my addr 172.21.15.57:0/4213407699 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:56.421 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1560bdb700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f155c153810 0x7f155c0c21e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:56.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1561bdd700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f155c159890 0x7f155c0c2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:56.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 -- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f155c159890 msgr2=0x7f155c0c2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:56.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f155c159890 0x7f155c0c2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 -- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f155c153810 msgr2=0x7f155c0c21e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:56.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f155c153810 0x7f155c0c21e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:56.422 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 -- 172.21.15.57:0/4213407699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0094f0 con 0x7f155c0c72a0 2023-06-23T01:19:56.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1561bdd700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f155c159890 0x7f155c0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:19:56.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15613dc700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c0c72a0 0x7f155c168dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f155800d890 tx=0x7f1558008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:56.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15517fa700 1 -- 172.21.15.57:0/4213407699 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15580125f0 con 0x7f155c0c72a0 2023-06-23T01:19:56.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15517fa700 1 -- 172.21.15.57:0/4213407699 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1558012750 con 0x7f155c0c72a0 2023-06-23T01:19:56.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f155c0c5b70 con 0x7f155c0c72a0 2023-06-23T01:19:56.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.414+0000 7f15517fa700 1 -- 172.21.15.57:0/4213407699 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1558012ad0 con 0x7f155c0c72a0 2023-06-23T01:19:56.423 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.415+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f155c0c6130 con 0x7f155c0c72a0 2023-06-23T01:19:56.424 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.415+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f155c064b50 con 0x7f155c0c72a0 2023-06-23T01:19:56.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.416+0000 7f15517fa700 1 -- 172.21.15.57:0/4213407699 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f155801e460 con 0x7f155c0c72a0 2023-06-23T01:19:56.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.416+0000 7f15517fa700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1540066ec0 0x7f1540069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:56.425 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.417+0000 7f1560bdb700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1540066ec0 0x7f1540069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:56.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.417+0000 7f15517fa700 1 -- 172.21.15.57:0/4213407699 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f15580849b0 con 0x7f155c0c72a0 2023-06-23T01:19:56.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.417+0000 7f1560bdb700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1540066ec0 0x7f1540069370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f155c159ce0 tx=0x7f154c016000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:56.426 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.421+0000 7f15517fa700 1 -- 172.21.15.57:0/4213407699 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f1558051d60 con 0x7f155c0c72a0 2023-06-23T01:19:56.617 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:56.614+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f155c15a370 con 0x7f155c0c72a0 2023-06-23T01:19:57.284 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.280+0000 7f15517fa700 1 -- 172.21.15.57:0/4213407699 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1558058e60 con 0x7f155c0c72a0 2023-06-23T01:19:57.294 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1540066ec0 msgr2=0x7f1540069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1540066ec0 0x7f1540069370 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f155c159ce0 tx=0x7f154c016000 comp rx=0 tx=0).stop 2023-06-23T01:19:57.295 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c0c72a0 msgr2=0x7f155c168dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c0c72a0 0x7f155c168dd0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f155800d890 tx=0x7f1558008040 comp rx=0 tx=0).stop 2023-06-23T01:19:57.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 shutdown_connections 2023-06-23T01:19:57.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f1540066ec0 0x7f1540069370 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.296 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f155c159890 0x7f155c0c2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f155c0c72a0 0x7f155c168dd0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 --2- 172.21.15.57:0/4213407699 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f155c153810 0x7f155c0c21e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 >> 172.21.15.57:0/4213407699 conn(0x7f155c070220 msgr2=0x7f155c067340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:57.297 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.289+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 shutdown_connections 2023-06-23T01:19:57.298 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.290+0000 7f1563640700 1 -- 172.21.15.57:0/4213407699 wait complete. 2023-06-23T01:19:57.298 INFO:teuthology.orchestra.run.smithi057.stderr:pool 'cephfs_data' created 2023-06-23T01:19:57.309 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-23T01:19:57.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.438+0000 7fec1d8fd700 1 Processor -- start 2023-06-23T01:19:57.442 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.438+0000 7fec1d8fd700 1 -- start start 2023-06-23T01:19:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec1d8fd700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7fec18157620 0x7fec180caed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec1d8fd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1815a600 0x7fec180cb410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec1d8fd700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180cdb00 con 0x7fec1815a230 2023-06-23T01:19:57.445 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec1d8fd700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec1815af00 con 0x7fec18157620 2023-06-23T01:19:57.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec1d8fd700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec1815b1e0 con 0x7fec1815a600 2023-06-23T01:19:57.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec16ffd700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7fec1815a230 0x7fec180cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44276/0 (socket says 172.21.15.57:44276) 2023-06-23T01:19:57.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec16ffd700 1 -- 172.21.15.57:0/2277844933 learned_addr learned my addr 172.21.15.57:0/2277844933 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:57.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec167fc700 1 --2- 172.21.15.57:0/2277844933 >> v2:172.21.15.123:3300/0 conn(0x7fec18157620 0x7fec180caed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec177fe700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1815a600 0x7fec180cb410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.446 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1701439919 0 0) 0x7fec08001290 con 0x7fec1815a230 2023-06-23T01:19:57.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec00003570 con 0x7fec1815a230 2023-06-23T01:19:57.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3435543514 0 0) 0x7fec00003570 con 0x7fec1815a230 2023-06-23T01:19:57.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1815a600 msgr2=0x7fec180cb410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:57.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1815a600 0x7fec180cb410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 >> v2:172.21.15.123:3300/0 conn(0x7fec18157620 msgr2=0x7fec180caed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:57.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 --2- 172.21.15.57:0/2277844933 >> v2:172.21.15.123:3300/0 conn(0x7fec18157620 0x7fec180caed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec1815b4c0 con 0x7fec1815a230 2023-06-23T01:19:57.447 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3756962719 0 0) 0x7fec08001780 con 0x7fec1815a230 2023-06-23T01:19:57.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.439+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 >> v1:172.21.15.57:6789/0 conn(0x7fec1815a230 legacy=0x7fec180cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1815a600 0x7fec00003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec000040b0 0x7fec00004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec000051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1815af00 con 0x7fec00004d50 2023-06-23T01:19:57.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec1815b1e0 con 0x7fec000040b0 2023-06-23T01:19:57.448 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0000bb60 con 0x7fec1815a600 2023-06-23T01:19:57.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec177fe700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec000051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec167fc700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec000040b0 0x7fec00004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec16ffd700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1815a600 0x7fec00003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec177fe700 1 -- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1815a600 msgr2=0x7fec00003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:57.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec177fe700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1815a600 0x7fec00003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.449 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec177fe700 1 -- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec000040b0 msgr2=0x7fec00004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:57.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec177fe700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec000040b0 0x7fec00004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec177fe700 1 -- 172.21.15.57:0/2277844933 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec00003570 con 0x7fec00004d50 2023-06-23T01:19:57.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec177fe700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec000051c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fec100094d0 tx=0x7fec1000b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0522e1a855ba536 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:57.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1000ea90 con 0x7fec00004d50 2023-06-23T01:19:57.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec1000f040 con 0x7fec00004d50 2023-06-23T01:19:57.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.440+0000 7fec15ffb700 1 -- 172.21.15.57:0/2277844933 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1000ee20 con 0x7fec00004d50 2023-06-23T01:19:57.450 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.441+0000 7fec1d8fd700 1 -- 172.21.15.57:0/2277844933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 msgr2=0x7fec000051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.441+0000 7fec1d8fd700 1 --2- 172.21.15.57:0/2277844933 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec000051c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fec100094d0 tx=0x7fec1000b040 comp rx=0 tx=0).stop 2023-06-23T01:19:57.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.441+0000 7fec177fe700 1 -- 172.21.15.57:0/2277844933 reap_dead start 2023-06-23T01:19:57.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.441+0000 7fec1d8fd700 1 -- 172.21.15.57:0/2277844933 shutdown_connections 2023-06-23T01:19:57.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.441+0000 7fec1d8fd700 1 -- 172.21.15.57:0/2277844933 >> 172.21.15.57:0/2277844933 conn(0x7fec18067340 msgr2=0x7fec18067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:57.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.441+0000 7fec1d8fd700 1 -- 172.21.15.57:0/2277844933 shutdown_connections 2023-06-23T01:19:57.451 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.441+0000 7fec1d8fd700 1 -- 172.21.15.57:0/2277844933 wait complete. 2023-06-23T01:19:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.442+0000 7fec1d8fd700 1 Processor -- start 2023-06-23T01:19:57.455 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.442+0000 7fec1d8fd700 1 -- start start 2023-06-23T01:19:57.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec1d8fd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec1814eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec1d8fd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec18156100 0x7fec180c88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.456 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec1d8fd700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1816a560 0x7fec180b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec1d8fd700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec18169870 con 0x7fec00004d50 2023-06-23T01:19:57.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec1d8fd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec180c9240 con 0x7fec18156100 2023-06-23T01:19:57.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec1d8fd700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec18169a80 con 0x7fec1816a560 2023-06-23T01:19:57.457 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec1814eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec1814eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34416/0 (socket says 172.21.15.57:34416) 2023-06-23T01:19:57.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 -- 172.21.15.57:0/1999982369 learned_addr learned my addr 172.21.15.57:0/1999982369 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:57.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec167fc700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec18156100 0x7fec180c88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 -- 172.21.15.57:0/1999982369 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1816a560 msgr2=0x7fec180b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1816a560 0x7fec180b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.458 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 -- 172.21.15.57:0/1999982369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec18156100 msgr2=0x7fec180c88d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec18156100 0x7fec180c88d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 -- 172.21.15.57:0/1999982369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec18169d60 con 0x7fec00004d50 2023-06-23T01:19:57.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.443+0000 7fec16ffd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec1814eae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fec080073a0 tx=0x7fec080079f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:57.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.444+0000 7febfeffd700 1 -- 172.21.15.57:0/1999982369 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0800f8d0 con 0x7fec00004d50 2023-06-23T01:19:57.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.444+0000 7febfeffd700 1 -- 172.21.15.57:0/1999982369 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0800fa50 con 0x7fec00004d50 2023-06-23T01:19:57.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.444+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec1816a040 con 0x7fec00004d50 2023-06-23T01:19:57.459 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.444+0000 7febfeffd700 1 -- 172.21.15.57:0/1999982369 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0800fe10 con 0x7fec00004d50 2023-06-23T01:19:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.444+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec180b18c0 con 0x7fec00004d50 2023-06-23T01:19:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.445+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec18064b50 con 0x7fec00004d50 2023-06-23T01:19:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.445+0000 7febfeffd700 1 -- 172.21.15.57:0/1999982369 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fec08013600 con 0x7fec00004d50 2023-06-23T01:19:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.445+0000 7febfeffd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fec04066e90 0x7fec04069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.445+0000 7febfeffd700 1 -- 172.21.15.57:0/1999982369 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5958+0+0 (secure 0 0 0) 0x7fec08084180 con 0x7fec00004d50 2023-06-23T01:19:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.445+0000 7fec167fc700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fec04066e90 0x7fec04069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.460 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.446+0000 7fec167fc700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fec04066e90 0x7fec04069340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fec18156db0 tx=0x7fec0c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:57.461 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.450+0000 7febfeffd700 1 -- 172.21.15.57:0/1999982369 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fec080513b0 con 0x7fec00004d50 2023-06-23T01:19:57.694 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.691+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fec180cdb00 con 0x7fec00004d50 2023-06-23T01:19:57.704 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.700+0000 7febfeffd700 1 -- 172.21.15.57:0/1999982369 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7fec180cdb00 con 0x7fec00004d50 2023-06-23T01:19:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.707+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fec04066e90 msgr2=0x7fec04069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.707+0000 7fec1d8fd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fec04066e90 0x7fec04069340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fec18156db0 tx=0x7fec0c008040 comp rx=0 tx=0).stop 2023-06-23T01:19:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.707+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 msgr2=0x7fec1814eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.711 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.707+0000 7fec1d8fd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec1814eae0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fec080073a0 tx=0x7fec080079f0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.710+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 shutdown_connections 2023-06-23T01:19:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.710+0000 7fec1d8fd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7fec04066e90 0x7fec04069340 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.710+0000 7fec1d8fd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7fec1816a560 0x7fec180b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.714 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.710+0000 7fec1d8fd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7fec00004d50 0x7fec1814eae0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.710+0000 7fec1d8fd700 1 --2- 172.21.15.57:0/1999982369 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fec18156100 0x7fec180c88d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.715 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.710+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 >> 172.21.15.57:0/1999982369 conn(0x7fec18067340 msgr2=0x7fec18158390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.712+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 shutdown_connections 2023-06-23T01:19:57.716 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.712+0000 7fec1d8fd700 1 -- 172.21.15.57:0/1999982369 wait complete. 2023-06-23T01:19:57.719 INFO:teuthology.orchestra.run.smithi057.stderr:new fs with metadata pool 3 and data pool 4 2023-06-23T01:19:57.737 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:19:57.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.855+0000 7f4ea3586700 1 Processor -- start 2023-06-23T01:19:57.859 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.855+0000 7f4ea3586700 1 -- start start 2023-06-23T01:19:57.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.856+0000 7f4ea3586700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4e9c159640 0x7f4e9c0b0db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.856+0000 7f4ea3586700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9c06c820 con 0x7f4e9c159270 2023-06-23T01:19:57.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.856+0000 7f4ea3586700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e9c0c8c80 con 0x7f4e9c06bbc0 2023-06-23T01:19:57.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.856+0000 7f4ea3586700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e9c068c00 con 0x7f4e9c159640 2023-06-23T01:19:57.862 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.856+0000 7f4ea1322700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f4e9c159270 0x7f4e9c06b4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44288/0 (socket says 172.21.15.57:44288) 2023-06-23T01:19:57.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.856+0000 7f4ea1322700 1 -- 172.21.15.57:0/186697639 learned_addr learned my addr 172.21.15.57:0/186697639 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:57.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.856+0000 7f4ea1b23700 1 --2- 172.21.15.57:0/186697639 >> v2:172.21.15.123:3300/0 conn(0x7f4e9c159640 0x7f4e9c0b0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.856+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1723273762 0 0) 0x7f4e9c06c820 con 0x7f4e9c159270 2023-06-23T01:19:57.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e88003570 con 0x7f4e9c159270 2023-06-23T01:19:57.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1064533665 0 0) 0x7f4e88003570 con 0x7f4e9c159270 2023-06-23T01:19:57.863 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 >> v2:172.21.15.123:3300/0 conn(0x7f4e9c159640 msgr2=0x7f4e9c0b0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:57.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 --2- 172.21.15.57:0/186697639 >> v2:172.21.15.123:3300/0 conn(0x7f4e9c159640 0x7f4e9c0b0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 >> v1:172.21.15.123:6789/0 conn(0x7f4e9c06bbc0 legacy=0x7f4e9c0af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 --> v1:172.21.15.57:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c068ee0 con 0x7f4e9c159270 2023-06-23T01:19:57.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 <== mon.0 v1:172.21.15.57:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3756962719 0 0) 0x7f4e90001470 con 0x7f4e9c159270 2023-06-23T01:19:57.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 >> v1:172.21.15.57:6789/0 conn(0x7f4e9c159270 legacy=0x7f4e9c06b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e88003a60 0x7f4e88003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.864 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e88004810 0x7f4e88004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e880051c0 0x7f4e8800ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c068c00 con 0x7f4e880051c0 2023-06-23T01:19:57.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8800c0d0 con 0x7f4e88003a60 2023-06-23T01:19:57.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8800c290 con 0x7f4e88004810 2023-06-23T01:19:57.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea1b23700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e880051c0 0x7f4e8800ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea1322700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e88003a60 0x7f4e88003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.865 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea0b21700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e88004810 0x7f4e88004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea1b23700 1 -- 172.21.15.57:0/186697639 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e88004810 msgr2=0x7f4e88004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:57.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea1b23700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e88004810 0x7f4e88004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea1b23700 1 -- 172.21.15.57:0/186697639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e88003a60 msgr2=0x7f4e88003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:57.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea1b23700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e88003a60 0x7f4e88003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea1b23700 1 -- 172.21.15.57:0/186697639 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e88003570 con 0x7f4e880051c0 2023-06-23T01:19:57.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.857+0000 7f4ea1b23700 1 -- 172.21.15.57:0/186697639 reap_dead start 2023-06-23T01:19:57.866 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4ea1b23700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e880051c0 0x7f4e8800ba60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400e210 tx=0x7f4e9400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3003844f2db96062 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:57.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400eac0 con 0x7f4e880051c0 2023-06-23T01:19:57.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9400f040 con 0x7f4e880051c0 2023-06-23T01:19:57.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4e8ffff700 1 -- 172.21.15.57:0/186697639 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9400ee70 con 0x7f4e880051c0 2023-06-23T01:19:57.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4ea3586700 1 -- 172.21.15.57:0/186697639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e880051c0 msgr2=0x7f4e8800ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4ea3586700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e880051c0 0x7f4e8800ba60 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4e9400e210 tx=0x7f4e9400b040 comp rx=0 tx=0).stop 2023-06-23T01:19:57.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4ea3586700 1 -- 172.21.15.57:0/186697639 shutdown_connections 2023-06-23T01:19:57.867 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4ea3586700 1 --2- 172.21.15.57:0/186697639 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e880051c0 0x7f4e8800ba60 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4ea3586700 1 -- 172.21.15.57:0/186697639 >> 172.21.15.57:0/186697639 conn(0x7f4e9c0cdfe0 msgr2=0x7f4e9c0ce3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:57.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4ea3586700 1 -- 172.21.15.57:0/186697639 shutdown_connections 2023-06-23T01:19:57.868 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.858+0000 7f4ea3586700 1 -- 172.21.15.57:0/186697639 wait complete. 2023-06-23T01:19:57.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.859+0000 7f4ea3586700 1 Processor -- start 2023-06-23T01:19:57.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.859+0000 7f4ea3586700 1 -- start start 2023-06-23T01:19:57.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.860+0000 7f4ea3586700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e880051c0 0x7f4e9c0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.870 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.860+0000 7f4ea3586700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9c0c4ff0 0x7f4e9c0c54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.860+0000 7f4ea3586700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c15b0d0 0x7f4e9c0c5a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.860+0000 7f4ea3586700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0c7c10 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.860+0000 7f4ea3586700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c15a3f0 con 0x7f4e880051c0 2023-06-23T01:19:57.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.860+0000 7f4ea3586700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c0c7930 con 0x7f4e9c0c4ff0 2023-06-23T01:19:57.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.860+0000 7f4ea1b23700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c15b0d0 0x7f4e9c0c5a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1b23700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c15b0d0 0x7f4e9c0c5a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34428/0 (socket says 172.21.15.57:34428) 2023-06-23T01:19:57.871 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1b23700 1 -- 172.21.15.57:0/2916258011 learned_addr learned my addr 172.21.15.57:0/2916258011 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:57.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1322700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e880051c0 0x7f4e9c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea0b21700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9c0c4ff0 0x7f4e9c0c54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1b23700 1 -- 172.21.15.57:0/2916258011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9c0c4ff0 msgr2=0x7f4e9c0c54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1b23700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9c0c4ff0 0x7f4e9c0c54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1b23700 1 -- 172.21.15.57:0/2916258011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e880051c0 msgr2=0x7f4e9c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:57.872 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1b23700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e880051c0 0x7f4e9c0c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:57.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1b23700 1 -- 172.21.15.57:0/2916258011 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c15a5f0 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea1b23700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c15b0d0 0x7f4e9c0c5a00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c15cfd0 tx=0x7f4e94019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:57.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4e8d7fa700 1 -- 172.21.15.57:0/2916258011 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9401d660 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4e8d7fa700 1 -- 172.21.15.57:0/2916258011 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e94022070 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e9c15a8d0 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4e8d7fa700 1 -- 172.21.15.57:0/2916258011 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9401da30 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.873 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.861+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e9c15c8f0 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.862+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e9c15add0 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.863+0000 7f4e8d7fa700 1 -- 172.21.15.57:0/2916258011 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f4e94003020 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.863+0000 7f4e8d7fa700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4e80066e90 0x7f4e80069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:57.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.863+0000 7f4e8d7fa700 1 -- 172.21.15.57:0/2916258011 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4e94024070 con 0x7f4e9c15b0d0 2023-06-23T01:19:57.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.863+0000 7f4ea1322700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4e80066e90 0x7f4e80069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:57.874 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.864+0000 7f4ea1322700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4e80066e90 0x7f4e80069340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e90001ce0 tx=0x7f4e90001e50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:57.875 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:57.868+0000 7f4e8d7fa700 1 -- 172.21.15.57:0/2916258011 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f4e94053d60 con 0x7f4e9c15b0d0 2023-06-23T01:19:58.065 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.062+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e9c064b50 con 0x7f4e9c15b0d0 2023-06-23T01:19:58.066 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.063+0000 7f4e8d7fa700 1 -- 172.21.15.57:0/2916258011 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16975 (secure 0 0 0) 0x7f4e9405ae60 con 0x7f4e9c15b0d0 2023-06-23T01:19:58.067 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:58.067 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":19,"fsid":"cbac5275-6376-4f14-97f5-8fe5f66396c8","created":"2023-06-23T01:19:39.130505+0000","modified":"2023-06-23T01:19:57.693067+0000","last_up_change":"2023-06-23T01:19:49.924346+0000","last_in_change":"2023-06-23T01:19:41.606376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T01:19:50.441526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T01:19:50.876930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:19:55.478594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T01:19:56.645137+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb40d155-cae0-4fda-b7af-ce312c958bbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":113976},{"type":"v1","addr":"172.21.15.57:6801","nonce":113976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":113976},{"type":"v1","addr":"172.21.15.57:6803","nonce":113976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":113976},{"type":"v1","addr":"172.21.15.57:6807","nonce":113976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":113976},{"type":"v1","addr":"172.21.15.57:6805","nonce":113976}]},"public_addr":"172.21.15.57:6801/113976","cluster_addr":"172.21.15.57:6803/113976","heartbeat_back_addr":"172.21.15.57:6807/113976","heartbeat_front_addr":"172.21.15.57:6805/113976","state":["exists","up"]},{"osd":1,"uuid":"e2dc7148-fc1a-48a4-8b06-013164077813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":113977},{"type":"v1","addr":"172.21.15.57:6809","nonce":113977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":113977},{"type":"v1","addr":"172.21.15.57:6811","nonce":113977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":113977},{"type":"v1","addr":"172.21.15.57:6815","nonce":113977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":113977},{"type":"v1","addr":"172.21.15.57:6813","nonce":113977}]},"public_addr":"172.21.15.57:6809/113977","cluster_addr":"172.21.15.57:6811/113977","heartbeat_back_addr":"172.21.15.57:6815/113977","heartbeat_front_addr":"172.21.15.57:6813/113977","state":["exists","up"]},{"osd":2,"uuid":"d30552e4-8c21-4ff9-baf7-07c7ed117bdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6817","nonce":113978},{"type":"v1","addr":"172.21.15.57:6819","nonce":113978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6821","nonce":113978},{"type":"v1","addr":"172.21.15.57:6823","nonce":113978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6829","nonce":113978},{"type":"v1","addr":"172.21.15.57:6831","nonce":113978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6825","nonce":113978},{"type":"v1","addr":"172.21.15.57:6827","nonce":113978}]},"public_addr":"172.21.15.57:6819/113978","cluster_addr":"172.21.15.57:6823/113978","heartbeat_back_addr":"172.21.15.57:6831/113978","heartbeat_front_addr":"172.21.15.57:6827/113978","state":["exists","up"]},{"osd":3,"uuid":"d22728ed-0946-49b6-871d-ab80ec2089c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":113995},{"type":"v1","addr":"172.21.15.57:6818","nonce":113995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":113995},{"type":"v1","addr":"172.21.15.57:6822","nonce":113995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":113995},{"type":"v1","addr":"172.21.15.57:6830","nonce":113995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":113995},{"type":"v1","addr":"172.21.15.57:6826","nonce":113995}]},"public_addr":"172.21.15.57:6818/113995","cluster_addr":"172.21.15.57:6822/113995","heartbeat_back_addr":"172.21.15.57:6830/113995","heartbeat_front_addr":"172.21.15.57:6826/113995","state":["exists","up"]},{"osd":4,"uuid":"a39e2e51-40b1-4c91-b045-f6f030dd6546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":114171},{"type":"v1","addr":"172.21.15.123:6801","nonce":114171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6803","nonce":114171},{"type":"v1","addr":"172.21.15.123:6805","nonce":114171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6811","nonce":114171},{"type":"v1","addr":"172.21.15.123:6813","nonce":114171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6807","nonce":114171},{"type":"v1","addr":"172.21.15.123:6809","nonce":114171}]},"public_addr":"172.21.15.123:6801/114171","cluster_addr":"172.21.15.123:6805/114171","heartbeat_back_addr":"172.21.15.123:6813/114171","heartbeat_front_addr":"172.21.15.123:6809/114171","state":["exists","up"]},{"osd":5,"uuid":"855f67e8-1b47-4fb7-a8c2-2f5df954a8ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":114172},{"type":"v1","addr":"172.21.15.123:6804","nonce":114172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":114172},{"type":"v1","addr":"172.21.15.123:6808","nonce":114172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":114172},{"type":"v1","addr":"172.21.15.123:6815","nonce":114172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":114172},{"type":"v1","addr":"172.21.15.123:6812","nonce":114172}]},"public_addr":"172.21.15.123:6804/114172","cluster_addr":"172.21.15.123:6808/114172","heartbeat_back_addr":"172.21.15.123:6815/114172","heartbeat_front_addr":"172.21.15.123:6812/114172","state":["exists","up"]},{"osd":6,"uuid":"a74edd8e-c135-473e-9394-f214d0405572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":114216},{"type":"v1","addr":"172.21.15.123:6825","nonce":114216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":114216},{"type":"v1","addr":"172.21.15.123:6827","nonce":114216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":114216},{"type":"v1","addr":"172.21.15.123:6831","nonce":114216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":114216},{"type":"v1","addr":"172.21.15.123:6829","nonce":114216}]},"public_addr":"172.21.15.123:6825/114216","cluster_addr":"172.21.15.123:6827/114216","heartbeat_back_addr":"172.21.15.123:6831/114216","heartbeat_front_addr":"172.21.15.123:6829/114216","state":["exists","up"]},{"osd":7,"uuid":"26725e66-b047-4e78-afc6-352a8ea70559","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":114173},{"type":"v1","addr":"172.21.15.123:6817","nonce":114173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":114173},{"type":"v1","addr":"172.21.15.123:6819","nonce":114173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":114173},{"type":"v1","addr":"172.21.15.123:6823","nonce":114173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":114173},{"type":"v1","addr":"172.21.15.123:6821","nonce":114173}]},"public_addr":"172.21.15.123:6817/114173","cluster_addr":"172.21.15.123:6819/114173","heartbeat_back_addr":"172.21.15.123:6823/114173","heartbeat_front_addr":"172.21.15.123:6821/114173","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.786276+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.749336+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.794142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.824374+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.812739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.708934+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.806842+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.766984+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:19:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4e80066e90 msgr2=0x7f4e80069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4e80066e90 0x7f4e80069340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4e90001ce0 tx=0x7f4e90001e50 comp rx=0 tx=0).stop 2023-06-23T01:19:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c15b0d0 msgr2=0x7f4e9c0c5a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c15b0d0 0x7f4e9c0c5a00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4e9c15cfd0 tx=0x7f4e94019000 comp rx=0 tx=0).stop 2023-06-23T01:19:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 shutdown_connections 2023-06-23T01:19:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4e80066e90 0x7f4e80069340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.069 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4e9c15b0d0 0x7f4e9c0c5a00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4e880051c0 0x7f4e9c0c7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 --2- 172.21.15.57:0/2916258011 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4e9c0c4ff0 0x7f4e9c0c54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 >> 172.21.15.57:0/2916258011 conn(0x7f4e9c0cdfe0 msgr2=0x7f4e9c0b35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:58.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 shutdown_connections 2023-06-23T01:19:58.070 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.065+0000 7f4ea3586700 1 -- 172.21.15.57:0/2916258011 wait complete. 2023-06-23T01:19:58.081 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:19:58.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.199+0000 7f052213e700 1 Processor -- start 2023-06-23T01:19:58.203 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.199+0000 7f052213e700 1 -- start start 2023-06-23T01:19:58.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f052213e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 0x7f051c1593c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.205 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f052213e700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f051c0cb210 0x7f051c15ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f052213e700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051c0d0190 con 0x7f051c0cf0c0 2023-06-23T01:19:58.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f052213e700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051c0cf580 con 0x7f051c0cb210 2023-06-23T01:19:58.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f052213e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c0cf860 con 0x7f051c158ff0 2023-06-23T01:19:58.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051affd700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7f051c0cf0c0 0x7f051c0d0620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44294/0 (socket says 172.21.15.57:44294) 2023-06-23T01:19:58.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051affd700 1 -- 172.21.15.57:0/730371238 learned_addr learned my addr 172.21.15.57:0/730371238 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:58.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051bfff700 1 --2- 172.21.15.57:0/730371238 >> v2:172.21.15.123:3300/0 conn(0x7f051c0cb210 0x7f051c15ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051b7fe700 1 --2- 172.21.15.57:0/730371238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 0x7f051c1593c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.206 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051a7fc700 1 -- 172.21.15.57:0/730371238 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 490765557 0 0) 0x7f051c0d0190 con 0x7f051c0cf0c0 2023-06-23T01:19:58.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051a7fc700 1 -- 172.21.15.57:0/730371238 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0508003570 con 0x7f051c0cf0c0 2023-06-23T01:19:58.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051bfff700 1 -- 172.21.15.57:0/730371238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 msgr2=0x7f051c1593c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:58.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051bfff700 1 --2- 172.21.15.57:0/730371238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 0x7f051c1593c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051bfff700 1 -- 172.21.15.57:0/730371238 >> v1:172.21.15.57:6789/0 conn(0x7f051c0cf0c0 legacy=0x7f051c0d0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.200+0000 7f051a7fc700 1 -- 172.21.15.57:0/730371238 <== mon.0 v1:172.21.15.57:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 369343818 0 0) 0x7f0508003570 con 0x7f051c0cf0c0 2023-06-23T01:19:58.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f051bfff700 1 -- 172.21.15.57:0/730371238 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051c0cfb40 con 0x7f051c0cb210 2023-06-23T01:19:58.207 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f051bfff700 1 --2- 172.21.15.57:0/730371238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 0x7f051c15ce90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0514009e60 tx=0x7f0514009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4e9bb15a64f7f99 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f051a7fc700 1 -- 172.21.15.57:0/730371238 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051400e070 con 0x7f051c0cb210 2023-06-23T01:19:58.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f051a7fc700 1 -- 172.21.15.57:0/730371238 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051400b7a0 con 0x7f051c0cb210 2023-06-23T01:19:58.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f051a7fc700 1 -- 172.21.15.57:0/730371238 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504001470 con 0x7f051c0cb210 2023-06-23T01:19:58.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f052213e700 1 -- 172.21.15.57:0/730371238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 msgr2=0x7f051c15ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f052213e700 1 --2- 172.21.15.57:0/730371238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 0x7f051c15ce90 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0514009e60 tx=0x7f0514009780 comp rx=0 tx=0).stop 2023-06-23T01:19:58.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f052213e700 1 -- 172.21.15.57:0/730371238 shutdown_connections 2023-06-23T01:19:58.208 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f052213e700 1 --2- 172.21.15.57:0/730371238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 0x7f051c15ce90 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f052213e700 1 --2- 172.21.15.57:0/730371238 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 0x7f051c1593c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f052213e700 1 -- 172.21.15.57:0/730371238 >> 172.21.15.57:0/730371238 conn(0x7f051c067340 msgr2=0x7f051c067740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:58.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f052213e700 1 -- 172.21.15.57:0/730371238 shutdown_connections 2023-06-23T01:19:58.209 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.201+0000 7f052213e700 1 -- 172.21.15.57:0/730371238 wait complete. 2023-06-23T01:19:58.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f052213e700 1 Processor -- start 2023-06-23T01:19:58.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f052213e700 1 -- start start 2023-06-23T01:19:58.211 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f052213e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 0x7f051c156e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f052213e700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0cf0c0 0x7f051c157a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f052213e700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 0x7f051c157fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f052213e700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c15a270 con 0x7f051c0cf0c0 2023-06-23T01:19:58.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f052213e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051c157360 con 0x7f051c0cb210 2023-06-23T01:19:58.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f052213e700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051c1593c0 con 0x7f051c158ff0 2023-06-23T01:19:58.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.203+0000 7f051affd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0cf0c0 0x7f051c157a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051affd700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0cf0c0 0x7f051c157a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34440/0 (socket says 172.21.15.57:34440) 2023-06-23T01:19:58.212 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051affd700 1 -- 172.21.15.57:0/594628484 learned_addr learned my addr 172.21.15.57:0/594628484 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:58.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051b7fe700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 0x7f051c156e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051bfff700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 0x7f051c157fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051affd700 1 -- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 msgr2=0x7f051c157fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051affd700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 0x7f051c157fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051affd700 1 -- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 msgr2=0x7f051c156e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051affd700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 0x7f051c156e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.213 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051affd700 1 -- 172.21.15.57:0/594628484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514009a20 con 0x7f051c0cf0c0 2023-06-23T01:19:58.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051b7fe700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 0x7f051c156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:19:58.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f051affd700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0cf0c0 0x7f051c157a90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f050400c8e0 tx=0x7f0504001a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f05137fe700 1 -- 172.21.15.57:0/594628484 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504005b30 con 0x7f051c0cf0c0 2023-06-23T01:19:58.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f05137fe700 1 -- 172.21.15.57:0/594628484 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0504005cb0 con 0x7f051c0cf0c0 2023-06-23T01:19:58.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051c159660 con 0x7f051c0cf0c0 2023-06-23T01:19:58.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f05137fe700 1 -- 172.21.15.57:0/594628484 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0504007340 con 0x7f051c0cf0c0 2023-06-23T01:19:58.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.204+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051c159c20 con 0x7f051c0cf0c0 2023-06-23T01:19:58.214 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.205+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051c15a420 con 0x7f051c0cf0c0 2023-06-23T01:19:58.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.206+0000 7f05137fe700 1 -- 172.21.15.57:0/594628484 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f050401b400 con 0x7f051c0cf0c0 2023-06-23T01:19:58.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.206+0000 7f05137fe700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f04fc066e90 0x7f04fc069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.206+0000 7f05137fe700 1 -- 172.21.15.57:0/594628484 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f0504084270 con 0x7f051c0cf0c0 2023-06-23T01:19:58.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.206+0000 7f051b7fe700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f04fc066e90 0x7f04fc069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.215 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.206+0000 7f051b7fe700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f04fc066e90 0x7f04fc069340 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f050c0113e0 tx=0x7f050c0091a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.216 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.210+0000 7f05137fe700 1 -- 172.21.15.57:0/594628484 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f051c15a420 con 0x7f051c0cf0c0 2023-06-23T01:19:58.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.406+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f051c15a420 con 0x7f051c0cf0c0 2023-06-23T01:19:58.410 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.407+0000 7f05137fe700 1 -- 172.21.15.57:0/594628484 <== mon.0 v2:172.21.15.57:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16975 (secure 0 0 0) 0x7f051c15a420 con 0x7f051c0cf0c0 2023-06-23T01:19:58.411 INFO:teuthology.orchestra.run.smithi057.stdout: 2023-06-23T01:19:58.411 INFO:teuthology.orchestra.run.smithi057.stdout:{"epoch":19,"fsid":"cbac5275-6376-4f14-97f5-8fe5f66396c8","created":"2023-06-23T01:19:39.130505+0000","modified":"2023-06-23T01:19:57.693067+0000","last_up_change":"2023-06-23T01:19:49.924346+0000","last_in_change":"2023-06-23T01:19:41.606376+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-06-23T01:19:50.441526+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":".mgr","create_time":"2023-06-23T01:19:50.876930+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:19:55.478594+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T01:19:56.645137+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"bb40d155-cae0-4fda-b7af-ce312c958bbf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6800","nonce":113976},{"type":"v1","addr":"172.21.15.57:6801","nonce":113976}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6802","nonce":113976},{"type":"v1","addr":"172.21.15.57:6803","nonce":113976}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6806","nonce":113976},{"type":"v1","addr":"172.21.15.57:6807","nonce":113976}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6804","nonce":113976},{"type":"v1","addr":"172.21.15.57:6805","nonce":113976}]},"public_addr":"172.21.15.57:6801/113976","cluster_addr":"172.21.15.57:6803/113976","heartbeat_back_addr":"172.21.15.57:6807/113976","heartbeat_front_addr":"172.21.15.57:6805/113976","state":["exists","up"]},{"osd":1,"uuid":"e2dc7148-fc1a-48a4-8b06-013164077813","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6808","nonce":113977},{"type":"v1","addr":"172.21.15.57:6809","nonce":113977}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6810","nonce":113977},{"type":"v1","addr":"172.21.15.57:6811","nonce":113977}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6814","nonce":113977},{"type":"v1","addr":"172.21.15.57:6815","nonce":113977}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6812","nonce":113977},{"type":"v1","addr":"172.21.15.57:6813","nonce":113977}]},"public_addr":"172.21.15.57:6809/113977","cluster_addr":"172.21.15.57:6811/113977","heartbeat_back_addr":"172.21.15.57:6815/113977","heartbeat_front_addr":"172.21.15.57:6813/113977","state":["exists","up"]},{"osd":2,"uuid":"d30552e4-8c21-4ff9-baf7-07c7ed117bdf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6817","nonce":113978},{"type":"v1","addr":"172.21.15.57:6819","nonce":113978}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6821","nonce":113978},{"type":"v1","addr":"172.21.15.57:6823","nonce":113978}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6829","nonce":113978},{"type":"v1","addr":"172.21.15.57:6831","nonce":113978}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6825","nonce":113978},{"type":"v1","addr":"172.21.15.57:6827","nonce":113978}]},"public_addr":"172.21.15.57:6819/113978","cluster_addr":"172.21.15.57:6823/113978","heartbeat_back_addr":"172.21.15.57:6831/113978","heartbeat_front_addr":"172.21.15.57:6827/113978","state":["exists","up"]},{"osd":3,"uuid":"d22728ed-0946-49b6-871d-ab80ec2089c6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6816","nonce":113995},{"type":"v1","addr":"172.21.15.57:6818","nonce":113995}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6820","nonce":113995},{"type":"v1","addr":"172.21.15.57:6822","nonce":113995}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6828","nonce":113995},{"type":"v1","addr":"172.21.15.57:6830","nonce":113995}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.57:6824","nonce":113995},{"type":"v1","addr":"172.21.15.57:6826","nonce":113995}]},"public_addr":"172.21.15.57:6818/113995","cluster_addr":"172.21.15.57:6822/113995","heartbeat_back_addr":"172.21.15.57:6830/113995","heartbeat_front_addr":"172.21.15.57:6826/113995","state":["exists","up"]},{"osd":4,"uuid":"a39e2e51-40b1-4c91-b045-f6f030dd6546","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6800","nonce":114171},{"type":"v1","addr":"172.21.15.123:6801","nonce":114171}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6803","nonce":114171},{"type":"v1","addr":"172.21.15.123:6805","nonce":114171}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6811","nonce":114171},{"type":"v1","addr":"172.21.15.123:6813","nonce":114171}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6807","nonce":114171},{"type":"v1","addr":"172.21.15.123:6809","nonce":114171}]},"public_addr":"172.21.15.123:6801/114171","cluster_addr":"172.21.15.123:6805/114171","heartbeat_back_addr":"172.21.15.123:6813/114171","heartbeat_front_addr":"172.21.15.123:6809/114171","state":["exists","up"]},{"osd":5,"uuid":"855f67e8-1b47-4fb7-a8c2-2f5df954a8ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6802","nonce":114172},{"type":"v1","addr":"172.21.15.123:6804","nonce":114172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6806","nonce":114172},{"type":"v1","addr":"172.21.15.123:6808","nonce":114172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6814","nonce":114172},{"type":"v1","addr":"172.21.15.123:6815","nonce":114172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6810","nonce":114172},{"type":"v1","addr":"172.21.15.123:6812","nonce":114172}]},"public_addr":"172.21.15.123:6804/114172","cluster_addr":"172.21.15.123:6808/114172","heartbeat_back_addr":"172.21.15.123:6815/114172","heartbeat_front_addr":"172.21.15.123:6812/114172","state":["exists","up"]},{"osd":6,"uuid":"a74edd8e-c135-473e-9394-f214d0405572","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6824","nonce":114216},{"type":"v1","addr":"172.21.15.123:6825","nonce":114216}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6826","nonce":114216},{"type":"v1","addr":"172.21.15.123:6827","nonce":114216}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6830","nonce":114216},{"type":"v1","addr":"172.21.15.123:6831","nonce":114216}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6828","nonce":114216},{"type":"v1","addr":"172.21.15.123:6829","nonce":114216}]},"public_addr":"172.21.15.123:6825/114216","cluster_addr":"172.21.15.123:6827/114216","heartbeat_back_addr":"172.21.15.123:6831/114216","heartbeat_front_addr":"172.21.15.123:6829/114216","state":["exists","up"]},{"osd":7,"uuid":"26725e66-b047-4e78-afc6-352a8ea70559","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6816","nonce":114173},{"type":"v1","addr":"172.21.15.123:6817","nonce":114173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6818","nonce":114173},{"type":"v1","addr":"172.21.15.123:6819","nonce":114173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6822","nonce":114173},{"type":"v1","addr":"172.21.15.123:6823","nonce":114173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:6820","nonce":114173},{"type":"v1","addr":"172.21.15.123:6821","nonce":114173}]},"public_addr":"172.21.15.123:6817/114173","cluster_addr":"172.21.15.123:6819/114173","heartbeat_back_addr":"172.21.15.123:6823/114173","heartbeat_front_addr":"172.21.15.123:6821/114173","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.786276+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.749336+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.794142+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.824374+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.812739+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.708934+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.806842+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:19:46.766984+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:19:58.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f04fc066e90 msgr2=0x7f04fc069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f04fc066e90 0x7f04fc069340 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f050c0113e0 tx=0x7f050c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.413 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0cf0c0 msgr2=0x7f051c157a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0cf0c0 0x7f051c157a90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f050400c8e0 tx=0x7f0504001a10 comp rx=0 tx=0).stop 2023-06-23T01:19:58.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 shutdown_connections 2023-06-23T01:19:58.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f04fc066e90 0x7f04fc069340 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f051c158ff0 0x7f051c157fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f051c0cb210 0x7f051c156e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 --2- 172.21.15.57:0/594628484 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f051c0cf0c0 0x7f051c157a90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.414 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 >> 172.21.15.57:0/594628484 conn(0x7f051c067340 msgr2=0x7f051c0c4fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:58.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 shutdown_connections 2023-06-23T01:19:58.415 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.409+0000 7f052213e700 1 -- 172.21.15.57:0/594628484 wait complete. 2023-06-23T01:19:58.425 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-23T01:19:58.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.546+0000 7ff47e778700 1 Processor -- start 2023-06-23T01:19:58.550 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.547+0000 7ff47e778700 1 -- start start 2023-06-23T01:19:58.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.547+0000 7ff47e778700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4781592b0 0x7ff478070be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.547+0000 7ff47e778700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7ff4780caaa0 0x7ff478068350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.547+0000 7ff47e778700 1 -- --> v1:172.21.15.57:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4780c7510 con 0x7ff478068890 2023-06-23T01:19:58.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.547+0000 7ff47e778700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff478068d10 con 0x7ff4780caaa0 2023-06-23T01:19:58.552 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.547+0000 7ff47e778700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff47806b7e0 con 0x7ff4781592b0 2023-06-23T01:19:58.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff47cd15700 1 --1- >> v1:172.21.15.57:6789/0 conn(0x7ff478068890 0x7ff478069ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.57:6789/0 says I am v1:172.21.15.57:44296/0 (socket says 172.21.15.57:44296) 2023-06-23T01:19:58.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff47cd15700 1 -- 172.21.15.57:0/1205704722 learned_addr learned my addr 172.21.15.57:0/1205704722 (peer_addr_for_me v1:172.21.15.57:0/0) 2023-06-23T01:19:58.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff477fff700 1 --2- 172.21.15.57:0/1205704722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4781592b0 0x7ff478070be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff4777fe700 1 --2- 172.21.15.57:0/1205704722 >> v2:172.21.15.123:3300/0 conn(0x7ff4780caaa0 0x7ff478068350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff476ffd700 1 -- 172.21.15.57:0/1205704722 <== mon.0 v1:172.21.15.57:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656757123 0 0) 0x7ff4780c7510 con 0x7ff478068890 2023-06-23T01:19:58.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff476ffd700 1 -- 172.21.15.57:0/1205704722 --> v1:172.21.15.57:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff460003570 con 0x7ff478068890 2023-06-23T01:19:58.553 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff4777fe700 1 -- 172.21.15.57:0/1205704722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4781592b0 msgr2=0x7ff478070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:58.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff4777fe700 1 --2- 172.21.15.57:0/1205704722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4781592b0 0x7ff478070be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff4777fe700 1 -- 172.21.15.57:0/1205704722 >> v1:172.21.15.57:6789/0 conn(0x7ff478068890 legacy=0x7ff478069ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff4777fe700 1 -- 172.21.15.57:0/1205704722 --> v2:172.21.15.123:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff47806bc70 con 0x7ff4780caaa0 2023-06-23T01:19:58.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff4777fe700 1 --2- 172.21.15.57:0/1205704722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4780caaa0 0x7ff478068350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0098e0 tx=0x7ff46c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ffc69d3011dd1a05 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff476ffd700 1 -- 172.21.15.57:0/1205704722 <== mon.1 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c011630 con 0x7ff4780caaa0 2023-06-23T01:19:58.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff476ffd700 1 -- 172.21.15.57:0/1205704722 <== mon.1 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff46c011790 con 0x7ff4780caaa0 2023-06-23T01:19:58.554 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.548+0000 7ff476ffd700 1 -- 172.21.15.57:0/1205704722 <== mon.1 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff46c011b30 con 0x7ff4780caaa0 2023-06-23T01:19:58.556 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.549+0000 7ff47e778700 1 -- 172.21.15.57:0/1205704722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4780caaa0 msgr2=0x7ff478068350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.549+0000 7ff47e778700 1 --2- 172.21.15.57:0/1205704722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4780caaa0 0x7ff478068350 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff46c0098e0 tx=0x7ff46c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.549+0000 7ff47e778700 1 -- 172.21.15.57:0/1205704722 shutdown_connections 2023-06-23T01:19:58.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.549+0000 7ff47e778700 1 --2- 172.21.15.57:0/1205704722 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff4781592b0 0x7ff478070be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.549+0000 7ff47e778700 1 --2- 172.21.15.57:0/1205704722 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4780caaa0 0x7ff478068350 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.549+0000 7ff47e778700 1 -- 172.21.15.57:0/1205704722 >> 172.21.15.57:0/1205704722 conn(0x7ff4780ca410 msgr2=0x7ff47806e110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:58.557 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.549+0000 7ff47e778700 1 -- 172.21.15.57:0/1205704722 shutdown_connections 2023-06-23T01:19:58.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.549+0000 7ff47e778700 1 -- 172.21.15.57:0/1205704722 wait complete. 2023-06-23T01:19:58.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.550+0000 7ff47e778700 1 Processor -- start 2023-06-23T01:19:58.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.550+0000 7ff47e778700 1 -- start start 2023-06-23T01:19:58.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47e778700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff478068890 0x7ff478150150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47e778700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4780caaa0 0x7ff4780b0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47e778700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4781592b0 0x7ff47816c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47e778700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780c4370 con 0x7ff4781592b0 2023-06-23T01:19:58.558 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47e778700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780b1140 con 0x7ff4780caaa0 2023-06-23T01:19:58.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47e778700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4780b1420 con 0x7ff478068890 2023-06-23T01:19:58.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47cd15700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4781592b0 0x7ff47816c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47cd15700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4781592b0 0x7ff47816c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34444/0 (socket says 172.21.15.57:34444) 2023-06-23T01:19:58.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47cd15700 1 -- 172.21.15.57:0/297287045 learned_addr learned my addr 172.21.15.57:0/297287045 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:58.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff4777fe700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4780caaa0 0x7ff4780b0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff477fff700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff478068890 0x7ff478150150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47cd15700 1 -- 172.21.15.57:0/297287045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff478068890 msgr2=0x7ff478150150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.559 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47cd15700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff478068890 0x7ff478150150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47cd15700 1 -- 172.21.15.57:0/297287045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4780caaa0 msgr2=0x7ff4780b0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47cd15700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff4780caaa0 0x7ff4780b0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff47cd15700 1 -- 172.21.15.57:0/297287045 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c009420 con 0x7ff4781592b0 2023-06-23T01:19:58.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.551+0000 7ff477fff700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7ff478068890 0x7ff478150150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:19:58.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.552+0000 7ff47cd15700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7ff4781592b0 0x7ff47816c9f0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7ff470001460 tx=0x7ff470001a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.552+0000 7ff45ffff700 1 -- 172.21.15.57:0/297287045 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47000a950 con 0x7ff4781592b0 2023-06-23T01:19:58.560 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.552+0000 7ff45ffff700 1 -- 172.21.15.57:0/297287045 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff47000aab0 con 0x7ff4781592b0 2023-06-23T01:19:58.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.552+0000 7ff47e778700 1 -- 172.21.15.57:0/297287045 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4780b1700 con 0x7ff4781592b0 2023-06-23T01:19:58.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.552+0000 7ff45ffff700 1 -- 172.21.15.57:0/297287045 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff47000ae30 con 0x7ff4781592b0 2023-06-23T01:19:58.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.552+0000 7ff47e778700 1 -- 172.21.15.57:0/297287045 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4780b19e0 con 0x7ff4781592b0 2023-06-23T01:19:58.561 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.553+0000 7ff47e778700 1 -- 172.21.15.57:0/297287045 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4780c44e0 con 0x7ff4781592b0 2023-06-23T01:19:58.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.554+0000 7ff45ffff700 1 -- 172.21.15.57:0/297287045 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7ff470007300 con 0x7ff4781592b0 2023-06-23T01:19:58.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.554+0000 7ff45ffff700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7ff464066e90 0x7ff464069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.554+0000 7ff45ffff700 1 -- 172.21.15.57:0/297287045 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6058+0+0 (secure 0 0 0) 0x7ff470085320 con 0x7ff4781592b0 2023-06-23T01:19:58.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.554+0000 7ff477fff700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7ff464066e90 0x7ff464069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.554+0000 7ff477fff700 1 --2- 172.21.15.57:0/297287045 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7ff464066e90 0x7ff464069340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff468006f00 tx=0x7ff468008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.562 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.557+0000 7ff45ffff700 1 -- 172.21.15.57:0/297287045 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7ff470052570 con 0x7ff4781592b0 2023-06-23T01:19:58.806 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 1 closest matches: 2023-06-23T01:19:58.806 INFO:teuthology.orchestra.run.smithi057.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:19:58.807 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2023-06-23T01:19:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:19:58.810 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-23T01:19:58.811 DEBUG:teuthology.orchestra.run.smithi057:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-23T01:19:58.935 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.932+0000 7f4c51f8a700 1 Processor -- start 2023-06-23T01:19:58.936 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.932+0000 7f4c51f8a700 1 -- start start 2023-06-23T01:19:58.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c51f8a700 1 --2- >> v2:172.21.15.123:3300/0 conn(0x7f4c4c06b760 0x7f4c4c0c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c51f8a700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c4c1583f0 0x7f4c4c1511b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c51f8a700 1 -- --> v1:172.21.15.123:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4c15b8d0 con 0x7f4c4c06bb30 2023-06-23T01:19:58.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c51f8a700 1 -- --> v2:172.21.15.57:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c15ad00 con 0x7f4c4c1583f0 2023-06-23T01:19:58.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c51f8a700 1 -- --> v2:172.21.15.123:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c15afa0 con 0x7f4c4c06b760 2023-06-23T01:19:58.938 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4affd700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c4c1583f0 0x7f4c4c1511b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4affd700 1 --2- >> v2:172.21.15.57:3300/0 conn(0x7f4c4c1583f0 0x7f4c4c1511b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34460/0 (socket says 172.21.15.57:34460) 2023-06-23T01:19:58.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4affd700 1 -- 172.21.15.57:0/2981155210 learned_addr learned my addr 172.21.15.57:0/2981155210 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:58.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4b7fe700 1 --2- 172.21.15.57:0/2981155210 >> v2:172.21.15.123:3300/0 conn(0x7f4c4c06b760 0x7f4c4c0c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4affd700 1 -- 172.21.15.57:0/2981155210 >> v2:172.21.15.123:3300/0 conn(0x7f4c4c06b760 msgr2=0x7f4c4c0c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:58.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4affd700 1 --2- 172.21.15.57:0/2981155210 >> v2:172.21.15.123:3300/0 conn(0x7f4c4c06b760 0x7f4c4c0c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4affd700 1 -- 172.21.15.57:0/2981155210 >> v1:172.21.15.123:6789/0 conn(0x7f4c4c06bb30 legacy=0x7f4c4c0c9940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:19:58.939 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4affd700 1 -- 172.21.15.57:0/2981155210 --> v2:172.21.15.57:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c15b280 con 0x7f4c4c1583f0 2023-06-23T01:19:58.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4affd700 1 --2- 172.21.15.57:0/2981155210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c4c1583f0 0x7f4c4c1511b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c06efe0 tx=0x7f4c40009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eabfe3e40d194ed6 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4a7fc700 1 -- 172.21.15.57:0/2981155210 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000b820 con 0x7f4c4c1583f0 2023-06-23T01:19:58.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4a7fc700 1 -- 172.21.15.57:0/2981155210 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c4000b980 con 0x7f4c4c1583f0 2023-06-23T01:19:58.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.933+0000 7f4c4a7fc700 1 -- 172.21.15.57:0/2981155210 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c4000bd20 con 0x7f4c4c1583f0 2023-06-23T01:19:58.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.934+0000 7f4c51f8a700 1 -- 172.21.15.57:0/2981155210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c4c1583f0 msgr2=0x7f4c4c1511b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.934+0000 7f4c51f8a700 1 --2- 172.21.15.57:0/2981155210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c4c1583f0 0x7f4c4c1511b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c06efe0 tx=0x7f4c40009250 comp rx=0 tx=0).stop 2023-06-23T01:19:58.940 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.934+0000 7f4c51f8a700 1 -- 172.21.15.57:0/2981155210 shutdown_connections 2023-06-23T01:19:58.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.934+0000 7f4c51f8a700 1 --2- 172.21.15.57:0/2981155210 >> v2:172.21.15.123:3300/0 conn(0x7f4c4c06b760 0x7f4c4c0c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.934+0000 7f4c51f8a700 1 --2- 172.21.15.57:0/2981155210 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c4c1583f0 0x7f4c4c1511b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.934+0000 7f4c51f8a700 1 -- 172.21.15.57:0/2981155210 >> 172.21.15.57:0/2981155210 conn(0x7f4c4c0cdfc0 msgr2=0x7f4c4c0ce3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:19:58.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.934+0000 7f4c51f8a700 1 -- 172.21.15.57:0/2981155210 shutdown_connections 2023-06-23T01:19:58.941 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.934+0000 7f4c51f8a700 1 -- 172.21.15.57:0/2981155210 wait complete. 2023-06-23T01:19:58.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.935+0000 7f4c51f8a700 1 Processor -- start 2023-06-23T01:19:58.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.935+0000 7f4c51f8a700 1 -- start start 2023-06-23T01:19:58.943 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c51f8a700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c4c06b760 0x7f4c4c0c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c51f8a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c06bb30 0x7f4c4c14cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c51f8a700 1 --2- >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c4c1583f0 0x7f4c4c0c76a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c51f8a700 1 -- --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0cb6f0 con 0x7f4c4c06b760 2023-06-23T01:19:58.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c51f8a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0cb260 con 0x7f4c4c06bb30 2023-06-23T01:19:58.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c51f8a700 1 -- --> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c0ca650 con 0x7f4c4c1583f0 2023-06-23T01:19:58.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c4c06b760 0x7f4c4c0c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.944 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4b7fe700 1 --2- >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c4c06b760 0x7f4c4c0c9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.57:3300/0 says I am v2:172.21.15.57:34472/0 (socket says 172.21.15.57:34472) 2023-06-23T01:19:58.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4b7fe700 1 -- 172.21.15.57:0/4076327853 learned_addr learned my addr 172.21.15.57:0/4076327853 (peer_addr_for_me v2:172.21.15.57:0/0) 2023-06-23T01:19:58.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4affd700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c06bb30 0x7f4c4c14cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4bfff700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c4c1583f0 0x7f4c4c0c76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4b7fe700 1 -- 172.21.15.57:0/4076327853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c4c1583f0 msgr2=0x7f4c4c0c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4b7fe700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c4c1583f0 0x7f4c4c0c76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4b7fe700 1 -- 172.21.15.57:0/4076327853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c06bb30 msgr2=0x7f4c4c14cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:19:58.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4b7fe700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c4c06bb30 0x7f4c4c14cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:19:58.945 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4b7fe700 1 -- 172.21.15.57:0/4076327853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c400094f0 con 0x7f4c4c06b760 2023-06-23T01:19:58.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.936+0000 7f4c4bfff700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.123:3301/0,v1:172.21.15.123:6790/0] conn(0x7f4c4c1583f0 0x7f4c4c0c76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:19:58.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.937+0000 7f4c4b7fe700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] conn(0x7f4c4c06b760 0x7f4c4c0c9d50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4c3c00d8c0 tx=0x7f4c3c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.937+0000 7f4c337fe700 1 -- 172.21.15.57:0/4076327853 <== mon.0 v2:172.21.15.57:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c0125f0 con 0x7f4c4c06b760 2023-06-23T01:19:58.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.937+0000 7f4c337fe700 1 -- 172.21.15.57:0/4076327853 <== mon.0 v2:172.21.15.57:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c3c012750 con 0x7f4c4c06b760 2023-06-23T01:19:58.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.937+0000 7f4c51f8a700 1 -- 172.21.15.57:0/4076327853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c0ca930 con 0x7f4c4c06b760 2023-06-23T01:19:58.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.937+0000 7f4c337fe700 1 -- 172.21.15.57:0/4076327853 <== mon.0 v2:172.21.15.57:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c3c012af0 con 0x7f4c4c06b760 2023-06-23T01:19:58.946 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.937+0000 7f4c51f8a700 1 -- 172.21.15.57:0/4076327853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c0ca290 con 0x7f4c4c06b760 2023-06-23T01:19:58.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.938+0000 7f4c51f8a700 1 -- 172.21.15.57:0/4076327853 --> [v2:172.21.15.57:3300/0,v1:172.21.15.57:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c0cb540 con 0x7f4c4c06b760 2023-06-23T01:19:58.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.938+0000 7f4c337fe700 1 -- 172.21.15.57:0/4076327853 <== mon.0 v2:172.21.15.57:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f4c3c01e460 con 0x7f4c4c06b760 2023-06-23T01:19:58.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.939+0000 7f4c337fe700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4c38066e90 0x7f4c38069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:19:58.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.939+0000 7f4c4affd700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4c38066e90 0x7f4c38069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:19:58.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.939+0000 7f4c337fe700 1 -- 172.21.15.57:0/4076327853 <== mon.0 v2:172.21.15.57:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f4c3c084b60 con 0x7f4c4c06b760 2023-06-23T01:19:58.947 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.939+0000 7f4c4affd700 1 --2- 172.21.15.57:0/4076327853 >> [v2:172.21.15.57:6832/113678,v1:172.21.15.57:6833/113678] conn(0x7f4c38066e90 0x7f4c38069340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4c4c06efe0 tx=0x7f4c40016000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:19:58.948 INFO:teuthology.orchestra.run.smithi057.stderr:2023-06-23T01:19:58.943+0000 7f4c337fe700 1 -- 172.21.15.57:0/4076327853 <== mon.0 v2:172.21.15.57:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f4c3c051d30 con 0x7f4c4c06b760 2023-06-23T01:19:59.190 INFO:teuthology.orchestra.run.smithi057.stderr:no valid command found; 1 closest matches: 2023-06-23T01:19:59.190 INFO:teuthology.orchestra.run.smithi057.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:19:59.190 INFO:teuthology.orchestra.run.smithi057.stderr:Error EINVAL: invalid command 2023-06-23T01:19:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:19:59.194 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:19:59.195 INFO:teuthology.misc:Shutting down mds daemons... 2023-06-23T01:19:59.196 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-23T01:19:59.197 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.205 INFO:tasks.ceph.mds.a:Stopped 2023-06-23T01:19:59.205 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-23T01:19:59.206 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.254 INFO:tasks.ceph.mds.c:Stopped 2023-06-23T01:19:59.255 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-23T01:19:59.255 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.265 INFO:tasks.ceph.mds.b:Stopped 2023-06-23T01:19:59.266 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-23T01:19:59.266 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.316 INFO:tasks.ceph.mds.d:Stopped 2023-06-23T01:19:59.316 INFO:teuthology.misc:Shutting down osd daemons... 2023-06-23T01:19:59.317 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-06-23T01:19:59.317 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.347 INFO:tasks.ceph.osd.0:Stopped 2023-06-23T01:19:59.348 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-06-23T01:19:59.349 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.417 INFO:tasks.ceph.osd.1:Stopped 2023-06-23T01:19:59.418 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-06-23T01:19:59.418 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.487 INFO:tasks.ceph.osd.2:Stopped 2023-06-23T01:19:59.487 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-06-23T01:19:59.488 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.555 INFO:tasks.ceph.osd.3:Stopped 2023-06-23T01:19:59.555 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-06-23T01:19:59.556 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.594 INFO:tasks.ceph.osd.4:Stopped 2023-06-23T01:19:59.596 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-06-23T01:19:59.596 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.664 INFO:tasks.ceph.osd.5:Stopped 2023-06-23T01:19:59.665 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-06-23T01:19:59.665 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.732 INFO:tasks.ceph.osd.6:Stopped 2023-06-23T01:19:59.733 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-06-23T01:19:59.733 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.811 INFO:tasks.ceph.osd.7:Stopped 2023-06-23T01:19:59.812 INFO:teuthology.misc:Shutting down mgr daemons... 2023-06-23T01:19:59.812 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-06-23T01:19:59.813 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.837 INFO:tasks.ceph.mgr.y:Stopped 2023-06-23T01:19:59.837 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-06-23T01:19:59.838 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.876 INFO:tasks.ceph.mgr.x:Stopped 2023-06-23T01:19:59.877 INFO:teuthology.misc:Shutting down mon daemons... 2023-06-23T01:19:59.878 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-06-23T01:19:59.878 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.888 INFO:tasks.ceph.mon.a:Stopped 2023-06-23T01:19:59.889 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-06-23T01:19:59.889 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.927 INFO:tasks.ceph.mon.b:Stopped 2023-06-23T01:19:59.928 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-06-23T01:19:59.928 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:19:59.978 INFO:tasks.ceph.mon.c:Stopped 2023-06-23T01:19:59.979 INFO:tasks.ceph:Checking cluster log for badness... 2023-06-23T01:19:59.979 DEBUG:teuthology.orchestra.run.smithi057:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2023-06-23T01:20:00.010 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:20:00.021 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:20:00.021 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-06-23T01:20:00.062 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:20:00.567 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:20:00.568 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-06-23T01:20:00.697 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:20:00.698 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-06-23T01:20:00.786 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:20:00.787 DEBUG:teuthology.orchestra.run.smithi057:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-06-23T01:20:00.861 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:20:00.862 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-06-23T01:20:01.152 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:20:01.153 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-06-23T01:20:01.226 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:20:01.227 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-06-23T01:20:01.327 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:20:01.328 DEBUG:teuthology.orchestra.run.smithi123:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-06-23T01:20:01.399 INFO:tasks.ceph:Archiving mon data... 2023-06-23T01:20:01.400 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/data/mon.a.tgz 2023-06-23T01:20:01.401 DEBUG:teuthology.orchestra.run.smithi057:> mktemp 2023-06-23T01:20:01.423 INFO:teuthology.orchestra.run.smithi057.stdout:/tmp/tmp.viNWHHgnJd 2023-06-23T01:20:01.424 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.viNWHHgnJd 2023-06-23T01:20:01.546 DEBUG:teuthology.orchestra.run.smithi057:> sudo chmod 0666 /tmp/tmp.viNWHHgnJd 2023-06-23T01:20:01.641 DEBUG:teuthology.orchestra.remote:smithi057:/tmp/tmp.viNWHHgnJd is 196KB 2023-06-23T01:20:01.694 DEBUG:teuthology.orchestra.run.smithi057:> rm -fr /tmp/tmp.viNWHHgnJd 2023-06-23T01:20:01.714 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/data/mon.b.tgz 2023-06-23T01:20:01.715 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-06-23T01:20:01.738 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.FaqbNn5dhh 2023-06-23T01:20:01.739 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.FaqbNn5dhh 2023-06-23T01:20:01.866 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.FaqbNn5dhh 2023-06-23T01:20:01.958 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.FaqbNn5dhh is 274KB 2023-06-23T01:20:02.051 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.FaqbNn5dhh 2023-06-23T01:20:02.069 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/data/mon.c.tgz 2023-06-23T01:20:02.070 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-06-23T01:20:02.126 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.4ZgtDYZOyj 2023-06-23T01:20:02.127 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.4ZgtDYZOyj 2023-06-23T01:20:02.237 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.4ZgtDYZOyj 2023-06-23T01:20:02.325 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.4ZgtDYZOyj is 274KB 2023-06-23T01:20:02.420 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.4ZgtDYZOyj 2023-06-23T01:20:02.437 INFO:tasks.ceph:Cleaning ceph cluster... 2023-06-23T01:20:02.438 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T01:20:02.440 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T01:20:02.503 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-06-23T01:20:02.504 DEBUG:teuthology.orchestra.run.smithi057:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T01:20:02.514 DEBUG:teuthology.orchestra.run.smithi123:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T01:20:02.546 INFO:teuthology.orchestra.run.smithi057.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T01:20:02.571 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T01:20:02.576 INFO:tasks.ceph:Archiving crash dumps... 2023-06-23T01:20:02.580 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/remote/smithi057/crash 2023-06-23T01:20:02.581 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T01:20:02.628 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/remote/smithi123/crash 2023-06-23T01:20:02.629 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T01:20:02.657 INFO:tasks.ceph:Shutting down logrotate 2023-06-23T01:20:02.658 DEBUG:teuthology.orchestra.run.smithi057:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:20:02.674 DEBUG:teuthology.orchestra.run.smithi123:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:20:02.746 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T01:20:02.779 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T01:20:02.806 INFO:tasks.ceph:Compressing logs... 2023-06-23T01:20:02.807 DEBUG:teuthology.orchestra.run.smithi057:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:20:02.821 DEBUG:teuthology.orchestra.run.smithi123:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:20:03.118 INFO:tasks.ceph:Archiving logs... 2023-06-23T01:20:03.118 DEBUG:teuthology.misc:Transferring archived files from smithi057:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/remote/smithi057/log 2023-06-23T01:20:03.120 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:20:03.204 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/remote/smithi123/log 2023-06-23T01:20:03.205 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:20:03.271 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:20:03.343 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=313e45e675e143a48305485b60e5c0e4 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi057 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:20:03.349 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-06-23T01:20:03.362 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-23T01:20:03.404 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-23T01:20:03.405 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:20:03.428 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:20:03.516 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-23T01:20:03.526 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-23T01:20:03.526 DEBUG:teuthology.orchestra.run.smithi057:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:20:03.529 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:20:03.545 INFO:teuthology.orchestra.run.smithi057.stderr:bash: ntpq: command not found 2023-06-23T01:20:03.546 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2023-06-23T01:20:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:20:03.550 INFO:teuthology.orchestra.run.smithi057.stdout:=============================================================================== 2023-06-23T01:20:03.551 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 31 +2066us[+2066us] +/- 95ms 2023-06-23T01:20:03.551 INFO:teuthology.orchestra.run.smithi057.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -1895us[-1895us] +/- 88ms 2023-06-23T01:20:03.552 INFO:teuthology.orchestra.run.smithi057.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 32 +11us[ +11us] +/- 66ms 2023-06-23T01:20:03.552 INFO:teuthology.orchestra.run.smithi057.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:20:03.554 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:20:03.554 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2023-06-23T01:20:03.555 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 29 +2069us[+2069us] +/- 95ms 2023-06-23T01:20:03.555 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 30 -1895us[-1862us] +/- 88ms 2023-06-23T01:20:03.556 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +7006ns[ +40us] +/- 66ms 2023-06-23T01:20:03.556 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:20:03.557 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-23T01:20:03.580 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-23T01:20:03.581 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-23T01:20:03.627 DEBUG:teuthology.orchestra.run.smithi057:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:20:03.723 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:20:03.810 DEBUG:teuthology.orchestra.run.smithi057:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:20:03.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:20:03.845 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:20:03.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:20:03.877 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-23T01:20:03.886 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-23T01:20:03.919 INFO:teuthology.task.internal:Duration was 1040.511702 seconds 2023-06-23T01:20:03.920 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-23T01:20:03.952 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-23T01:20:03.953 DEBUG:teuthology.orchestra.run.smithi057:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:20:03.956 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:20:04.003 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:20:04.004 INFO:teuthology.orchestra.run.smithi057.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:20:04.297 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-23T01:20:04.298 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi057.front.sepia.ceph.com 2023-06-23T01:20:04.298 DEBUG:teuthology.orchestra.run.smithi057:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:20:04.325 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2023-06-23T01:20:04.326 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:20:04.349 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-23T01:20:04.350 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:20:04.368 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:20:04.427 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-23T01:20:04.428 DEBUG:teuthology.orchestra.run.smithi057:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:20:04.449 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:20:04.543 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-23T01:20:04.560 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-23T01:20:04.561 DEBUG:teuthology.orchestra.run.smithi057:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:20:04.590 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:20:04.617 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-23T01:20:04.633 DEBUG:teuthology.orchestra.run.smithi057:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:20:04.636 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:20:04.662 INFO:teuthology.orchestra.run.smithi057.stdout:kernel.core_pattern = core 2023-06-23T01:20:04.684 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2023-06-23T01:20:04.706 DEBUG:teuthology.orchestra.run.smithi057:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:20:04.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:20:04.744 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:20:04.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:20:04.763 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-23T01:20:04.777 INFO:teuthology.task.internal:Transferring archived files... 2023-06-23T01:20:04.778 DEBUG:teuthology.misc:Transferring archived files from smithi057:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/remote/smithi057 2023-06-23T01:20:04.779 DEBUG:teuthology.orchestra.run.smithi057:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:20:04.822 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939/remote/smithi123 2023-06-23T01:20:04.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:20:04.865 INFO:teuthology.task.internal:Removing archive directory... 2023-06-23T01:20:04.866 DEBUG:teuthology.orchestra.run.smithi057:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:20:04.868 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:20:04.912 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-23T01:20:04.924 INFO:teuthology.task.internal:Not uploading archives. 2023-06-23T01:20:04.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-23T01:20:04.952 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-23T01:20:04.952 DEBUG:teuthology.orchestra.run.smithi057:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:20:04.954 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:20:04.970 INFO:teuthology.orchestra.run.smithi057.stdout: 525338 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:20 /home/ubuntu/cephtest 2023-06-23T01:20:04.972 INFO:teuthology.orchestra.run.smithi123.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:20 /home/ubuntu/cephtest 2023-06-23T01:20:04.973 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-23T01:20:04.984 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-23T01:20:05.072 INFO:teuthology.nuke:Checking targets against current locks 2023-06-23T01:20:05.100 DEBUG:teuthology.nuke:shortname: smithi057 2023-06-23T01:20:05.101 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:20:05.125 DEBUG:teuthology.nuke:shortname: smithi123 2023-06-23T01:20:05.126 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:20:05.130 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi057.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:53:57.066027', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDpmFKNFbg8LfVlJA1uLOjbxgiw4+0fk092Q7BVjQpvTvl6JuPfSD6R0l/eOW+NQaS56gZ/rfd94TfJ8AEa86TmkfMTja/kU9VUsjKaHQuZUxAD27+I+p7anHddByJwo4BpNBmGMdqcJjJRmsBw/m81tdx0lIvubMm/9WxPGJuAMbYrWkiKvLuSEKIgPUwPJOJDgq3+XIpD2Fbm4HWnzq/Lc546n+53S1O+yBK6o9Mg4mgcQoC0ePsEG03+J43LEuie2P5SG3d6smDBaCzuHFrBt1hK9Qg2OA/SoXDus/pnea8JslBKWziXKy5pN1oBfSssmfjVGHN4XYT/HRFprap/IjmCcYBqZUxFX+MrNF9iEqSpuBbQejFhIwZDHjP1bi7Jjt3pJi0xhpX6e212ovUA6fPqhH18m7hNY+iVuYkgYO44N1qTigooLRHg62G/vfSy1vondDQxeqhkQNHMEGk44AIbrEEDa/spGSpfJpZmNBdrmkuAvsuZa7PKh26zZ+k='} 2023-06-23T01:20:05.151 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311939', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 00:53:57.064491', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOO/uyCNxlFTHtgGYzIK80IutIj5fa7IQdy3A5VOOeCfVDWpZeDUdsZg/EgE6A5K+LT9cJR/J03iCupr5cnYRzE='} 2023-06-23T01:20:05.185 INFO:teuthology.orchestra.console:Power off smithi057 2023-06-23T01:20:05.187 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:20:05.220 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:20:05.239 INFO:teuthology.orchestra.console:Power off smithi123 2023-06-23T01:20:05.240 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:20:05.263 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:20:09.276 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:20:09.287 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:20:13.392 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:20:13.403 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:20:13.614 INFO:teuthology.orchestra.console:Power off for smithi123 completed 2023-06-23T01:20:17.508 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi057.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:20:17.721 INFO:teuthology.orchestra.console:Power off for smithi057 completed 2023-06-23T01:20:17.826 INFO:teuthology.run:Summary data: description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/snapshot}} duration: 1040.5117018222809 failure_reason: 'Command failed on smithi057 with status 22: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=313e45e675e143a48305485b60e5c0e4 status: fail success: false 2023-06-23T01:20:17.827 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T01:20:17.908 INFO:teuthology.run:FAIL