2023-06-23T01:17:48.136 INFO:root:teuthology version: 0.0.1.dev128+g076bbeb 2023-06-23T01:17:48.137 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} --archive /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967 --verbose --owner scheduled_yuriw@teuthology --name yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi -- /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/orig.config.yaml 2023-06-23T01:17:48.182 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T01:17:48.274 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967 branch: wip-yuri4-testing-2023-06-21-0846-quincy description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} email: yweinste@redhat.com first_in_suite: false job_id: '7311967' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.4' overrides: admin_socket: branch: wip-yuri4-testing-2023-06-21-0846-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-06-21-0846-quincy sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 9037 sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c sleep_before_teardown: 0 subset: 111/120000 suite: fs suite_branch: wip-yuri4-testing-2023-06-21-0846-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c targets: smithi092.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyCtGo6+Kvz2SWXvHcKq7TwgALZE7O2ZdSSDCKapdOYVDM/+QueBF/SOjjHzoK5xE3qRycvt4MeThoeNOlqGjBDuBnQ1xGPKdvQk852T9BVxz4/2FXr4bBQkf3meLqHxBufyzi07l7SA2ix1Vt6cZKib1QD9uAQL2chWtiTBItmBT16F2fP5qTm46PP+qRVTce+SWbH6QzOXCXiW8Td4zJurxt7VUlWBOVGh8dndG5QnoBLSndUMyiRI+dz34yhUPtRiWXhpDFQXO0Fx3psirsHTyRLczUunFKD3/TpRhcztV4LefNT22EheSHzUazFylz0kl0G9kf4KerisAgZPlJnV9288oXEQDZsgJ3MdZYX9YYcH66DFaveR2jfFYXTBlZ88DEVlQX8o+ThLVFc+K/UwNVlkzBNfsXrkLt+2fYXSgOem9cEuVRue/aG1b4e6Z/Gs2c4UQrwPTk3qA1M8RgQaWMDxq8xtrzs3upVTXBbHJrdjGjbZuVj+/jOeXVyZ8= smithi119.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbatMXeLcMLuQDmdjxHrNuwug4CM7BnUzrhJ0MzSXJQPq5T+21eDfm3gFGTmd8j1sLTQGFLEZdVxC5LMsbJg4z5BLpBBtuwaFKNxM54ePZPOFwXTQx8uUC8aY/xJEecFwDcaIYjlzuQF7zedT+8ZQLWWoUOKmyOqq/VHZIyU0LXNy1yWR3ZSX0Y+L0AjXXlRB85AwxoXlR0mIwXmeT39ozkK/jqz+FDtlnh6mVAf9zpsdl8jKXcMRFKU8uAE4ByOfSiQFkkKIt5FXkGFkc55IvLuodTRi7eB/h08ZeL+ApFKjeOP9WZjNB+mMfLHKBwTBoNl3hYaQZWGPAShxgfeG0tqysLYKVcRfAleAvafOpSWBoaf7JLuhAcZDv1eshFfC98fkIJFlahh0xmzyzwFKXk3zlwmzn+EmOhpUCiYIsksb+6gSz78RsQjAZMFrZukMct8DTaJwqh+gujOG1LEvh4r1jiIvQGokgwAQ9DYQ5ULPE2MqdkpEe9hW7x7npks0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - kclient: null - cephfs_test_runner: fail_on_skip: false modules: - tasks.cephfs.test_volumes.TestVolumes - tasks.cephfs.test_volumes.TestSubvolumeGroups - tasks.cephfs.test_volumes.TestSubvolumes - tasks.cephfs.test_subvolume.TestSubvolume teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 076bbebc42a14f7d568aaa78eabb0038327bcb23 timestamp: 2023-06-22_15:25:25 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.23459 2023-06-23T01:17:48.274 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa; will attempt to use it 2023-06-23T01:17:48.275 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks 2023-06-23T01:17:48.275 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-06-23T01:17:48.277 INFO:teuthology.task.internal:Checking packages... 2023-06-23T01:17:48.307 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash '42ddc46e6e6582d8e29a5a982a81886fe028be2c' 2023-06-23T01:17:48.307 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-06-23T01:17:48.307 INFO:teuthology.packaging:ref: None 2023-06-23T01:17:48.308 INFO:teuthology.packaging:tag: None 2023-06-23T01:17:48.308 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T01:17:48.308 INFO:teuthology.packaging:sha1: 42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:17:48.309 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-06-21-0846-quincy 2023-06-23T01:17:48.444 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-719.g42ddc46e 2023-06-23T01:17:48.445 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-06-23T01:17:48.452 INFO:teuthology.task.internal:no buildpackages task found 2023-06-23T01:17:48.452 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-06-23T01:17:48.465 INFO:teuthology.task.internal:Saving configuration 2023-06-23T01:17:48.486 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-06-23T01:17:48.493 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:17:48.522 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 01:12:30.125762', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPPMHp85ru2jyFE31Qri5XPllwyjN8OPHFj6ZGJB+vGzP+ArzZC3B0Se8pPVZtGh5x2z7JbIS0jjcjXhXlfl2vg='} 2023-06-23T01:17:48.547 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 01:12:30.127509', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDa3ht/VX6LTjui9Dl0RaX5tq9mHmeXaF5HScdX3J85v/k2glIUs+foCnV/WO5ZdSlQBaKjyyZgS5h81Q+me074='} 2023-06-23T01:17:48.548 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-06-23T01:17:48.557 INFO:teuthology.task.internal:roles: ubuntu@smithi092.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-06-23T01:17:48.557 INFO:teuthology.task.internal:roles: ubuntu@smithi119.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-06-23T01:17:48.558 INFO:teuthology.run_tasks:Running task console_log... 2023-06-23T01:17:48.641 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fad2163b040>, signals=[15]) 2023-06-23T01:17:48.642 INFO:teuthology.run_tasks:Running task internal.connect... 2023-06-23T01:17:48.649 INFO:teuthology.task.internal:Opening connections... 2023-06-23T01:17:48.650 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:17:48.652 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:17:48.725 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:17:48.726 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:17:48.832 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-06-23T01:17:48.839 DEBUG:teuthology.orchestra.run.smithi092:> uname -m 2023-06-23T01:17:48.856 INFO:teuthology.orchestra.run.smithi092.stdout:x86_64 2023-06-23T01:17:48.857 DEBUG:teuthology.orchestra.run.smithi092:> cat /etc/os-release 2023-06-23T01:17:48.912 INFO:teuthology.orchestra.run.smithi092.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T01:17:48.912 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T01:17:48.912 INFO:teuthology.orchestra.run.smithi092.stdout:ID="rhel" 2023-06-23T01:17:48.913 INFO:teuthology.orchestra.run.smithi092.stdout:ID_LIKE="fedora" 2023-06-23T01:17:48.913 INFO:teuthology.orchestra.run.smithi092.stdout:VERSION_ID="8.4" 2023-06-23T01:17:48.913 INFO:teuthology.orchestra.run.smithi092.stdout:PLATFORM_ID="platform:el8" 2023-06-23T01:17:48.913 INFO:teuthology.orchestra.run.smithi092.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T01:17:48.913 INFO:teuthology.orchestra.run.smithi092.stdout:ANSI_COLOR="0;31" 2023-06-23T01:17:48.913 INFO:teuthology.orchestra.run.smithi092.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T01:17:48.913 INFO:teuthology.orchestra.run.smithi092.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T01:17:48.914 INFO:teuthology.orchestra.run.smithi092.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T01:17:48.914 INFO:teuthology.orchestra.run.smithi092.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T01:17:48.914 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:17:48.914 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T01:17:48.914 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T01:17:48.914 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T01:17:48.914 INFO:teuthology.orchestra.run.smithi092.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T01:17:48.915 INFO:teuthology.lock.ops:Updating smithi092.front.sepia.ceph.com on lock server 2023-06-23T01:17:48.938 DEBUG:teuthology.orchestra.run.smithi119:> uname -m 2023-06-23T01:17:48.954 INFO:teuthology.orchestra.run.smithi119.stdout:x86_64 2023-06-23T01:17:48.955 DEBUG:teuthology.orchestra.run.smithi119:> cat /etc/os-release 2023-06-23T01:17:49.010 INFO:teuthology.orchestra.run.smithi119.stdout:NAME="Red Hat Enterprise Linux" 2023-06-23T01:17:49.010 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION="8.4 (Ootpa)" 2023-06-23T01:17:49.011 INFO:teuthology.orchestra.run.smithi119.stdout:ID="rhel" 2023-06-23T01:17:49.011 INFO:teuthology.orchestra.run.smithi119.stdout:ID_LIKE="fedora" 2023-06-23T01:17:49.011 INFO:teuthology.orchestra.run.smithi119.stdout:VERSION_ID="8.4" 2023-06-23T01:17:49.011 INFO:teuthology.orchestra.run.smithi119.stdout:PLATFORM_ID="platform:el8" 2023-06-23T01:17:49.011 INFO:teuthology.orchestra.run.smithi119.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.4 (Ootpa)" 2023-06-23T01:17:49.011 INFO:teuthology.orchestra.run.smithi119.stdout:ANSI_COLOR="0;31" 2023-06-23T01:17:49.011 INFO:teuthology.orchestra.run.smithi119.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8.4:GA" 2023-06-23T01:17:49.012 INFO:teuthology.orchestra.run.smithi119.stdout:HOME_URL="https://www.redhat.com/" 2023-06-23T01:17:49.012 INFO:teuthology.orchestra.run.smithi119.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-06-23T01:17:49.012 INFO:teuthology.orchestra.run.smithi119.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-06-23T01:17:49.012 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:17:49.012 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-06-23T01:17:49.012 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.4 2023-06-23T01:17:49.012 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-06-23T01:17:49.013 INFO:teuthology.orchestra.run.smithi119.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.4" 2023-06-23T01:17:49.013 INFO:teuthology.lock.ops:Updating smithi119.front.sepia.ceph.com on lock server 2023-06-23T01:17:49.038 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-06-23T01:17:49.053 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-06-23T01:17:49.060 INFO:teuthology.task.internal:Checking for old test directory... 2023-06-23T01:17:49.061 DEBUG:teuthology.orchestra.run.smithi092:> test '!' -e /home/ubuntu/cephtest 2023-06-23T01:17:49.064 DEBUG:teuthology.orchestra.run.smithi119:> test '!' -e /home/ubuntu/cephtest 2023-06-23T01:17:49.078 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-06-23T01:17:49.085 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-06-23T01:17:49.085 DEBUG:teuthology.orchestra.run.smithi092:> test -z $(ls -A /var/lib/ceph) 2023-06-23T01:17:49.119 DEBUG:teuthology.orchestra.run.smithi119:> test -z $(ls -A /var/lib/ceph) 2023-06-23T01:17:49.133 INFO:teuthology.orchestra.run.smithi092.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T01:17:49.134 INFO:teuthology.orchestra.run.smithi119.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2023-06-23T01:17:49.136 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-06-23T01:17:49.197 INFO:teuthology.run_tasks:Running task kernel... 2023-06-23T01:17:49.213 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-06-23T01:17:49.214 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-06-23T01:17:49.214 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-06-23T01:17:49.215 DEBUG:teuthology.orchestra.run.smithi092:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T01:17:49.215 DEBUG:teuthology.orchestra.run.smithi119:> test -f /run/.containerenv -o -f /.dockerenv 2023-06-23T01:17:49.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:17:49.231 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2023-06-23T01:17:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:17:49.233 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2023-06-23T01:17:49.287 INFO:teuthology.orchestra.run.smithi092.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T01:17:49.287 INFO:teuthology.task.kernel:Running kernel on smithi092: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T01:17:49.287 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum install -y kernel 2023-06-23T01:17:49.289 INFO:teuthology.orchestra.run.smithi119.stdout:4.18.0-305.25.1.el8_4.x86_64 2023-06-23T01:17:49.289 INFO:teuthology.task.kernel:Running kernel on smithi119: 4.18.0-305.25.1.el8_4.x86_64 2023-06-23T01:17:49.289 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum install -y kernel 2023-06-23T01:17:50.659 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2023-06-23T01:17:50.660 INFO:teuthology.orchestra.run.smithi119.stdout:Unable to read consumer identity 2023-06-23T01:17:50.715 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T01:17:50.715 INFO:teuthology.orchestra.run.smithi092.stdout:Unable to read consumer identity 2023-06-23T01:17:54.328 INFO:teuthology.orchestra.run.smithi092.stdout:Copr repo for python3-asyncssh owned by ceph 1.5 kB/s | 3.5 kB 00:02 2023-06-23T01:17:54.437 INFO:teuthology.orchestra.run.smithi119.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T01:17:56.762 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 6.6 MB/s | 16 MB 00:02 2023-06-23T01:17:56.784 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 16 MB 00:02 2023-06-23T01:18:00.024 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-06-23T01:18:00.026 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 4.7 MB/s | 24 kB 00:00 2023-06-23T01:18:00.749 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T01:18:00.750 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T01:18:00.750 INFO:teuthology.orchestra.run.smithi119.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T01:18:00.760 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-4.18.0-80.el8.x86_64 is already installed. 2023-06-23T01:18:00.760 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-4.18.0-425.3.1.el8.x86_64 is already installed. 2023-06-23T01:18:00.761 INFO:teuthology.orchestra.run.smithi092.stdout:Package kernel-4.18.0-305.25.1.el8_4.x86_64 is already installed. 2023-06-23T01:18:00.907 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2023-06-23T01:18:00.908 INFO:teuthology.orchestra.run.smithi119.stdout:Nothing to do. 2023-06-23T01:18:00.908 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2023-06-23T01:18:00.910 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-06-23T01:18:00.911 INFO:teuthology.orchestra.run.smithi092.stdout:Nothing to do. 2023-06-23T01:18:00.911 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-06-23T01:18:01.031 DEBUG:teuthology.orchestra.run.smithi092:> echo no | sudo yum reinstall kernel || true 2023-06-23T01:18:01.045 DEBUG:teuthology.orchestra.run.smithi119:> echo no | sudo yum reinstall kernel || true 2023-06-23T01:18:01.376 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T01:18:01.377 INFO:teuthology.orchestra.run.smithi092.stdout:Unable to read consumer identity 2023-06-23T01:18:01.389 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2023-06-23T01:18:01.389 INFO:teuthology.orchestra.run.smithi119.stdout:Unable to read consumer identity 2023-06-23T01:18:01.665 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:01 ago on Fri 23 Jun 2023 01:18:00 AM UTC. 2023-06-23T01:18:01.677 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:01 ago on Fri 23 Jun 2023 01:18:00 AM UTC. 2023-06-23T01:18:01.825 INFO:teuthology.orchestra.run.smithi119.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T01:18:01.828 INFO:teuthology.orchestra.run.smithi119.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:18:01.828 INFO:teuthology.orchestra.run.smithi119.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:18:01.847 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T01:18:01.847 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:18:01.847 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:18:01.858 INFO:teuthology.orchestra.run.smithi119.stderr:Error: No packages marked for reinstall. 2023-06-23T01:18:01.873 INFO:teuthology.orchestra.run.smithi092.stderr:Error: No packages marked for reinstall. 2023-06-23T01:18:01.913 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum reinstall -y kernel || true 2023-06-23T01:18:01.922 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum reinstall -y kernel || true 2023-06-23T01:18:02.257 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2023-06-23T01:18:02.257 INFO:teuthology.orchestra.run.smithi119.stdout:Unable to read consumer identity 2023-06-23T01:18:02.262 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T01:18:02.262 INFO:teuthology.orchestra.run.smithi092.stdout:Unable to read consumer identity 2023-06-23T01:18:02.537 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 01:18:00 AM UTC. 2023-06-23T01:18:02.542 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:00:02 ago on Fri 23 Jun 2023 01:18:00 AM UTC. 2023-06-23T01:18:02.687 INFO:teuthology.orchestra.run.smithi119.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T01:18:02.687 INFO:teuthology.orchestra.run.smithi119.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:18:02.687 INFO:teuthology.orchestra.run.smithi119.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:18:02.694 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-80.el8.x86_64 (from anaconda) not available. 2023-06-23T01:18:02.694 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-425.3.1.el8.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:18:02.694 INFO:teuthology.orchestra.run.smithi092.stdout:Installed package kernel-4.18.0-305.25.1.el8_4.x86_64 (from rhel-8-for-x86_64-baseos-rpms) not available. 2023-06-23T01:18:02.702 INFO:teuthology.orchestra.run.smithi119.stderr:Error: No packages marked for reinstall. 2023-06-23T01:18:02.705 INFO:teuthology.orchestra.run.smithi092.stderr:Error: No packages marked for reinstall. 2023-06-23T01:18:02.751 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:18:02.755 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:18:02.875 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:02.876 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:02.876 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:02.876 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T01:18:02.876 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T01:18:02.876 INFO:teuthology.task.kernel:Installing distro kernel on client.1... 2023-06-23T01:18:02.877 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi119.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T01:18:02.877 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:18:02.880 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:02.881 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:02.881 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-305.25.1.el8_4.x86_64 Expected: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:02.881 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-06-23T01:18:02.881 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-06-23T01:18:02.882 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2023-06-23T01:18:02.882 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi092.front.sepia.ceph.com, path=None, version=distro) 2023-06-23T01:18:02.882 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q kernel | sort -rV | head -n 1 2023-06-23T01:18:03.002 INFO:teuthology.orchestra.run.smithi119.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:03.003 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:03.003 DEBUG:teuthology.orchestra.run.smithi119:> sudo rpm -qi grub2-tools 2023-06-23T01:18:03.005 INFO:teuthology.orchestra.run.smithi092.stdout:kernel-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:03.007 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:03.007 DEBUG:teuthology.orchestra.run.smithi092:> sudo rpm -qi grub2-tools 2023-06-23T01:18:03.030 INFO:teuthology.orchestra.run.smithi119.stdout:Name : grub2-tools 2023-06-23T01:18:03.031 INFO:teuthology.orchestra.run.smithi119.stdout:Epoch : 1 2023-06-23T01:18:03.031 INFO:teuthology.orchestra.run.smithi119.stdout:Version : 2.02 2023-06-23T01:18:03.031 INFO:teuthology.orchestra.run.smithi119.stdout:Release : 99.el8_4.1 2023-06-23T01:18:03.031 INFO:teuthology.orchestra.run.smithi119.stdout:Architecture: x86_64 2023-06-23T01:18:03.031 INFO:teuthology.orchestra.run.smithi119.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T01:18:03.032 INFO:teuthology.orchestra.run.smithi119.stdout:Group : System Environment/Base 2023-06-23T01:18:03.032 INFO:teuthology.orchestra.run.smithi119.stdout:Size : 9264631 2023-06-23T01:18:03.032 INFO:teuthology.orchestra.run.smithi119.stdout:License : GPLv3+ 2023-06-23T01:18:03.032 INFO:teuthology.orchestra.run.smithi119.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T01:18:03.032 INFO:teuthology.orchestra.run.smithi119.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T01:18:03.032 INFO:teuthology.orchestra.run.smithi119.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T01:18:03.032 INFO:teuthology.orchestra.run.smithi119.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T01:18:03.033 INFO:teuthology.orchestra.run.smithi119.stdout:Relocations : (not relocatable) 2023-06-23T01:18:03.033 INFO:teuthology.orchestra.run.smithi119.stdout:Packager : Red Hat, Inc. 2023-06-23T01:18:03.033 INFO:teuthology.orchestra.run.smithi119.stdout:Vendor : Red Hat, Inc. 2023-06-23T01:18:03.033 INFO:teuthology.orchestra.run.smithi119.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T01:18:03.033 INFO:teuthology.orchestra.run.smithi119.stdout:Summary : Support tools for GRUB. 2023-06-23T01:18:03.033 INFO:teuthology.orchestra.run.smithi119.stdout:Description : 2023-06-23T01:18:03.033 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:18:03.033 INFO:teuthology.orchestra.run.smithi119.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T01:18:03.034 INFO:teuthology.orchestra.run.smithi119.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T01:18:03.034 INFO:teuthology.orchestra.run.smithi119.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T01:18:03.034 INFO:teuthology.orchestra.run.smithi119.stdout:hardware devices. 2023-06-23T01:18:03.034 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:18:03.034 INFO:teuthology.orchestra.run.smithi119.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T01:18:03.035 INFO:teuthology.orchestra.run.smithi092.stdout:Name : grub2-tools 2023-06-23T01:18:03.035 INFO:teuthology.orchestra.run.smithi092.stdout:Epoch : 1 2023-06-23T01:18:03.035 INFO:teuthology.orchestra.run.smithi092.stdout:Version : 2.02 2023-06-23T01:18:03.035 INFO:teuthology.orchestra.run.smithi092.stdout:Release : 99.el8_4.1 2023-06-23T01:18:03.035 INFO:teuthology.orchestra.run.smithi092.stdout:Architecture: x86_64 2023-06-23T01:18:03.035 INFO:teuthology.orchestra.run.smithi092.stdout:Install Date: Wed 21 Dec 2022 01:02:32 PM UTC 2023-06-23T01:18:03.036 INFO:teuthology.orchestra.run.smithi092.stdout:Group : System Environment/Base 2023-06-23T01:18:03.036 INFO:teuthology.orchestra.run.smithi092.stdout:Size : 9264631 2023-06-23T01:18:03.036 INFO:teuthology.orchestra.run.smithi092.stdout:License : GPLv3+ 2023-06-23T01:18:03.036 INFO:teuthology.orchestra.run.smithi092.stdout:Signature : RSA/SHA256, Thu 09 Sep 2021 09:41:55 AM UTC, Key ID 199e2f91fd431d51 2023-06-23T01:18:03.036 INFO:teuthology.orchestra.run.smithi092.stdout:Source RPM : grub2-2.02-99.el8_4.1.src.rpm 2023-06-23T01:18:03.036 INFO:teuthology.orchestra.run.smithi092.stdout:Build Date : Tue 18 May 2021 07:46:34 AM UTC 2023-06-23T01:18:03.036 INFO:teuthology.orchestra.run.smithi092.stdout:Build Host : x86-vm-07.build.eng.bos.redhat.com 2023-06-23T01:18:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Relocations : (not relocatable) 2023-06-23T01:18:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Packager : Red Hat, Inc. 2023-06-23T01:18:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Vendor : Red Hat, Inc. 2023-06-23T01:18:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:URL : http://www.gnu.org/software/grub/ 2023-06-23T01:18:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Summary : Support tools for GRUB. 2023-06-23T01:18:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:Description : 2023-06-23T01:18:03.037 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:18:03.037 INFO:teuthology.orchestra.run.smithi092.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-06-23T01:18:03.038 INFO:teuthology.orchestra.run.smithi092.stdout:customizable bootloader with modular architecture. It supports a rich 2023-06-23T01:18:03.038 INFO:teuthology.orchestra.run.smithi092.stdout:variety of kernel formats, file systems, computer architectures and 2023-06-23T01:18:03.038 INFO:teuthology.orchestra.run.smithi092.stdout:hardware devices. 2023-06-23T01:18:03.038 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:18:03.038 INFO:teuthology.orchestra.run.smithi092.stdout:This subpackage provides tools for support of all platforms. 2023-06-23T01:18:03.039 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T01:18:03.039 INFO:teuthology.task.kernel:Updating grub on smithi119 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:03.039 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T01:18:03.040 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-06-23T01:18:03.040 INFO:teuthology.task.kernel:Updating grub on smithi092 to boot 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:03.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-06-23T01:18:04.172 INFO:teuthology.orchestra.run.smithi092.stderr:Generating grub configuration file ... 2023-06-23T01:18:04.338 INFO:teuthology.orchestra.run.smithi119.stderr:Generating grub configuration file ... 2023-06-23T01:18:05.993 INFO:teuthology.orchestra.run.smithi092.stderr:done 2023-06-23T01:18:05.995 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2023-06-23T01:18:06.018 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.ZHvD1dKeQf 2023-06-23T01:18:06.018 DEBUG:teuthology.orchestra.run.smithi092:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZHvD1dKeQf 2023-06-23T01:18:06.087 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.ZHvD1dKeQf 2023-06-23T01:18:06.252 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.ZHvD1dKeQf is 6KB 2023-06-23T01:18:06.270 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.ZHvD1dKeQf 2023-06-23T01:18:06.285 DEBUG:teuthology.orchestra.run.smithi092:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T01:18:06.292 INFO:teuthology.orchestra.run.smithi119.stderr:done 2023-06-23T01:18:06.294 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2023-06-23T01:18:06.308 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.TbXIIC9Jzw 2023-06-23T01:18:06.308 DEBUG:teuthology.orchestra.run.smithi119:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TbXIIC9Jzw 2023-06-23T01:18:06.351 INFO:teuthology.orchestra.run.smithi092.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T01:18:06.351 INFO:teuthology.orchestra.run.smithi092.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T01:18:06.351 INFO:teuthology.orchestra.run.smithi092.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T01:18:06.351 INFO:teuthology.orchestra.run.smithi092.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T01:18:06.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:06.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.TbXIIC9Jzw 2023-06-23T01:18:06.515 DEBUG:teuthology.orchestra.run.smithi092:> sudo shutdown -r now 2023-06-23T01:18:06.535 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.TbXIIC9Jzw is 6KB 2023-06-23T01:18:06.583 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.TbXIIC9Jzw 2023-06-23T01:18:06.596 DEBUG:teuthology.orchestra.run.smithi119:> sudo /bin/ls /boot/loader/entries || true 2023-06-23T01:18:06.668 INFO:teuthology.orchestra.run.smithi119.stdout:298e228f3b5e401e8ac0f5ae159a2e06-0-rescue.conf 2023-06-23T01:18:06.669 INFO:teuthology.orchestra.run.smithi119.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-305.25.1.el8_4.x86_64.conf 2023-06-23T01:18:06.669 INFO:teuthology.orchestra.run.smithi119.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64.conf 2023-06-23T01:18:06.669 INFO:teuthology.orchestra.run.smithi119.stdout:298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-80.el8.x86_64.conf 2023-06-23T01:18:06.670 DEBUG:teuthology.orchestra.run.smithi119:> sudo grub2-set-default 298e228f3b5e401e8ac0f5ae159a2e06-4.18.0-425.3.1.el8.x86_64 2023-06-23T01:18:06.820 DEBUG:teuthology.orchestra.run.smithi119:> sudo shutdown -r now 2023-06-23T01:18:36.544 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-06-23T01:18:36.545 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T01:18:36.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:18:36.824 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-06-23T01:18:36.824 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-06-23T01:18:36.825 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:18:55.088 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-06-23T01:18:55.348 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2023-06-23T01:19:25.116 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T01:19:25.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:19:25.349 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-06-23T01:19:25.350 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:19:43.472 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.92 2023-06-23T01:19:43.728 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.119 2023-06-23T01:20:13.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T01:20:13.501 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:20:13.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-06-23T01:20:13.730 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:20:13.889 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-06-23T01:20:14.019 DEBUG:teuthology.orchestra.run.smithi119:> true 2023-06-23T01:20:14.174 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T01:20:14.175 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T01:20:14.175 DEBUG:teuthology.orchestra.run.smithi092:> uname -r 2023-06-23T01:20:14.190 INFO:teuthology.orchestra.run.smithi092.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T01:20:14.190 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:20:14.191 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T01:20:14.191 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T01:20:14.310 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-06-23T01:20:14.311 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "4.18.0-425.3.1.el8.x86_64"... 2023-06-23T01:20:14.311 DEBUG:teuthology.orchestra.run.smithi119:> uname -r 2023-06-23T01:20:14.327 INFO:teuthology.orchestra.run.smithi119.stdout:4.18.0-425.3.1.el8.x86_64 2023-06-23T01:20:14.327 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-425.3.1.el8.x86_64 vs 4.18.0-425.3.1.el8.x86_64 2023-06-23T01:20:14.328 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-06-23T01:20:14.328 DEBUG:teuthology.task.kernel:Distro of this test job: rhel 2023-06-23T01:20:15.192 DEBUG:teuthology.parallel:result is None 2023-06-23T01:20:15.329 DEBUG:teuthology.parallel:result is None 2023-06-23T01:20:15.329 INFO:teuthology.run_tasks:Running task internal.base... 2023-06-23T01:20:15.337 INFO:teuthology.task.internal:Creating test directory... 2023-06-23T01:20:15.337 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T01:20:15.340 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-06-23T01:20:15.357 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-06-23T01:20:15.364 INFO:teuthology.run_tasks:Running task internal.archive... 2023-06-23T01:20:15.374 INFO:teuthology.task.internal:Creating archive directory... 2023-06-23T01:20:15.374 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T01:20:15.396 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-06-23T01:20:15.436 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-06-23T01:20:15.446 INFO:teuthology.task.internal:Enabling coredump saving... 2023-06-23T01:20:15.447 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T01:20:15.469 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-06-23T01:20:15.550 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:20:15.570 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:20:15.577 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:20:15.594 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-06-23T01:20:15.596 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-06-23T01:20:15.605 INFO:teuthology.task.internal:Configuring sudo... 2023-06-23T01:20:15.606 DEBUG:teuthology.orchestra.run.smithi092:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T01:20:15.609 DEBUG:teuthology.orchestra.run.smithi119:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-06-23T01:20:15.635 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-06-23T01:20:15.645 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-06-23T01:20:15.645 DEBUG:teuthology.orchestra.run.smithi092:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T01:20:15.675 DEBUG:teuthology.orchestra.run.smithi119:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-06-23T01:20:15.691 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:20:15.774 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:20:15.812 DEBUG:teuthology.orchestra.run.smithi092:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:20:15.898 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:20:15.933 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:20:15.933 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T01:20:15.997 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:20:16.045 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-06-23T01:20:16.085 DEBUG:teuthology.orchestra.run.smithi119:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:20:16.170 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-06-23T01:20:16.207 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:20:16.208 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-06-23T01:20:16.272 DEBUG:teuthology.orchestra.run.smithi092:> sudo service rsyslog restart 2023-06-23T01:20:16.274 DEBUG:teuthology.orchestra.run.smithi119:> sudo service rsyslog restart 2023-06-23T01:20:16.327 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:20:16.357 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:20:16.684 INFO:teuthology.run_tasks:Running task internal.timer... 2023-06-23T01:20:16.694 INFO:teuthology.task.internal:Starting timer... 2023-06-23T01:20:16.694 INFO:teuthology.run_tasks:Running task pcp... 2023-06-23T01:20:16.709 INFO:teuthology.run_tasks:Running task selinux... 2023-06-23T01:20:16.722 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-06-23T01:20:16.722 DEBUG:teuthology.orchestra.run.smithi092:> sudo service auditd rotate 2023-06-23T01:20:16.783 INFO:teuthology.orchestra.run.smithi092.stdout:Rotating logs: [ OK ] 2023-06-23T01:20:16.785 DEBUG:teuthology.orchestra.run.smithi119:> sudo service auditd rotate 2023-06-23T01:20:16.864 INFO:teuthology.orchestra.run.smithi119.stdout:Rotating logs: [ OK ] 2023-06-23T01:20:16.865 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-06-23T01:20:16.866 DEBUG:teuthology.orchestra.run.smithi092:> /usr/sbin/getenforce 2023-06-23T01:20:16.890 INFO:teuthology.orchestra.run.smithi092.stdout:Permissive 2023-06-23T01:20:16.890 DEBUG:teuthology.orchestra.run.smithi119:> /usr/sbin/getenforce 2023-06-23T01:20:16.909 INFO:teuthology.orchestra.run.smithi119.stdout:Permissive 2023-06-23T01:20:16.910 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi092.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi119.front.sepia.ceph.com': 'permissive'} 2023-06-23T01:20:16.910 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:20:16.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:20:16.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:20:16.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:20:16.985 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-06-23T01:20:16.985 DEBUG:teuthology.orchestra.run.smithi092:> sudo /usr/sbin/setenforce permissive 2023-06-23T01:20:17.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo /usr/sbin/setenforce permissive 2023-06-23T01:20:17.051 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-06-23T01:20:17.062 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-06-23T01:20:17.062 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-06-23T01:20:17.083 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-06-23T01:20:17.084 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi092.front.sepia.ceph.com,smithi119.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-06-23T01:28:29.251 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi092.front.sepia.ceph.com'), Remote(name='ubuntu@smithi119.front.sepia.ceph.com')] 2023-06-23T01:28:29.253 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi092.front.sepia.ceph.com' 2023-06-23T01:28:29.254 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi092.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:28:29.324 DEBUG:teuthology.orchestra.run.smithi092:> true 2023-06-23T01:28:29.386 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi119.front.sepia.ceph.com' 2023-06-23T01:28:29.387 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi119.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-06-23T01:28:29.452 DEBUG:teuthology.orchestra.run.smithi119:> true 2023-06-23T01:28:29.523 INFO:teuthology.run_tasks:Running task clock... 2023-06-23T01:28:29.537 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-06-23T01:28:29.537 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T01:28:29.538 DEBUG:teuthology.orchestra.run.smithi092:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:28:29.540 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-06-23T01:28:29.541 DEBUG:teuthology.orchestra.run.smithi119:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:28:29.566 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T01:28:29.580 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T01:28:29.594 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-06-23T01:28:29.608 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-06-23T01:28:29.632 INFO:teuthology.orchestra.run.smithi092.stderr:sudo: ntpd: command not found 2023-06-23T01:28:29.642 INFO:teuthology.orchestra.run.smithi092.stdout:506 Cannot talk to daemon 2023-06-23T01:28:29.649 INFO:teuthology.orchestra.run.smithi119.stderr:sudo: ntpd: command not found 2023-06-23T01:28:29.656 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T01:28:29.659 INFO:teuthology.orchestra.run.smithi119.stdout:506 Cannot talk to daemon 2023-06-23T01:28:29.669 INFO:teuthology.orchestra.run.smithi092.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T01:28:29.673 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-06-23T01:28:29.687 INFO:teuthology.orchestra.run.smithi119.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-06-23T01:28:29.748 INFO:teuthology.orchestra.run.smithi092.stderr:bash: ntpq: command not found 2023-06-23T01:28:29.750 INFO:teuthology.orchestra.run.smithi092.stdout:210 Number of sources = 4 2023-06-23T01:28:29.750 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:28:29.750 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-06-23T01:28:29.751 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:28:29.751 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:28:29.751 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:28:29.751 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:28:29.751 INFO:teuthology.orchestra.run.smithi119.stderr:bash: ntpq: command not found 2023-06-23T01:28:29.753 INFO:teuthology.orchestra.run.smithi119.stdout:210 Number of sources = 4 2023-06-23T01:28:29.753 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:28:29.753 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2023-06-23T01:28:29.754 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:28:29.754 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:28:29.754 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:28:29.754 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:28:29.755 INFO:teuthology.run_tasks:Running task install... 2023-06-23T01:28:29.768 DEBUG:teuthology.task.install:project ceph 2023-06-23T01:28:29.768 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}} 2023-06-23T01:28:29.769 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'} 2023-06-23T01:28:29.769 INFO:teuthology.task.install:Using flavor: default 2023-06-23T01:28:29.774 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-06-23T01:28:29.775 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-06-23T01:28:29.775 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T01:28:29.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:28:29.778 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'tag': None, 'wait_for_package': False} 2023-06-23T01:28:29.778 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:28:29.914 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T01:28:29.914 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T01:28:29.995 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/ 2023-06-23T01:28:29.995 INFO:teuthology.task.install.rpm:Package version is 17.2.6-719.g42ddc46e 2023-06-23T01:28:30.030 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T01:28:30.031 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:28:30.032 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T01:28:30.059 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T01:28:30.059 DEBUG:teuthology.orchestra.run.smithi119:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T01:28:30.099 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-06-21-0846-quincy/42ddc46e6e6582d8e29a5a982a81886fe028be2c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-06-23T01:28:30.100 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:28:30.100 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-06-23T01:28:30.126 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel on remote rpm x86_64 2023-06-23T01:28:30.126 DEBUG:teuthology.orchestra.run.smithi092:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/42ddc46e6e6582d8e29a5a982a81886fe028be2c/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-06-23T01:28:30.127 DEBUG:teuthology.orchestra.run.smithi119:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T01:28:30.191 DEBUG:teuthology.orchestra.run.smithi092:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-06-23T01:28:30.203 DEBUG:teuthology.orchestra.run.smithi119:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T01:28:30.265 DEBUG:teuthology.orchestra.run.smithi092:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-06-23T01:28:30.267 INFO:teuthology.orchestra.run.smithi119.stdout:check_obsoletes = 1 2023-06-23T01:28:30.268 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum clean all 2023-06-23T01:28:30.328 INFO:teuthology.orchestra.run.smithi092.stdout:check_obsoletes = 1 2023-06-23T01:28:30.330 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum clean all 2023-06-23T01:28:30.657 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2023-06-23T01:28:30.721 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T01:28:30.964 INFO:teuthology.orchestra.run.smithi119.stdout:44 files removed 2023-06-23T01:28:31.014 DEBUG:teuthology.orchestra.run.smithi119:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T01:28:31.098 INFO:teuthology.orchestra.run.smithi092.stdout:44 files removed 2023-06-23T01:28:31.148 DEBUG:teuthology.orchestra.run.smithi092:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel 2023-06-23T01:28:31.361 INFO:teuthology.orchestra.run.smithi119.stdout:Updating Subscription Management repositories. 2023-06-23T01:28:31.499 INFO:teuthology.orchestra.run.smithi092.stdout:Updating Subscription Management repositories. 2023-06-23T01:28:32.073 INFO:teuthology.orchestra.run.smithi119.stdout:ceph packages for x86_64 348 kB/s | 78 kB 00:00 2023-06-23T01:28:32.264 INFO:teuthology.orchestra.run.smithi092.stdout:ceph packages for x86_64 252 kB/s | 78 kB 00:00 2023-06-23T01:28:32.297 INFO:teuthology.orchestra.run.smithi119.stdout:ceph noarch packages 50 kB/s | 9.2 kB 00:00 2023-06-23T01:28:32.570 INFO:teuthology.orchestra.run.smithi119.stdout:ceph source packages 5.9 kB/s | 1.4 kB 00:00 2023-06-23T01:28:32.605 INFO:teuthology.orchestra.run.smithi092.stdout:ceph noarch packages 30 kB/s | 9.2 kB 00:00 2023-06-23T01:28:32.880 INFO:teuthology.orchestra.run.smithi092.stdout:ceph source packages 5.9 kB/s | 1.4 kB 00:00 2023-06-23T01:28:35.018 INFO:teuthology.orchestra.run.smithi119.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T01:28:35.334 INFO:teuthology.orchestra.run.smithi092.stdout:Copr repo for python3-asyncssh owned by ceph 1.4 kB/s | 3.5 kB 00:02 2023-06-23T01:28:39.731 INFO:teuthology.orchestra.run.smithi119.stdout:Extra Packages for Enterprise Linux 3.3 MB/s | 16 MB 00:04 2023-06-23T01:28:42.988 INFO:teuthology.orchestra.run.smithi119.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2023-06-23T01:28:43.373 INFO:teuthology.orchestra.run.smithi119.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 125 MB/s | 38 MB 00:00 2023-06-23T01:28:48.841 INFO:teuthology.orchestra.run.smithi119.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 121 MB/s | 33 MB 00:00 2023-06-23T01:28:51.497 INFO:teuthology.orchestra.run.smithi092.stdout:Extra Packages for Enterprise Linux 991 kB/s | 16 MB 00:16 2023-06-23T01:28:54.755 INFO:teuthology.orchestra.run.smithi092.stdout:lab-extras 3.8 MB/s | 24 kB 00:00 2023-06-23T01:28:55.124 INFO:teuthology.orchestra.run.smithi092.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 115 MB/s | 33 MB 00:00 2023-06-23T01:28:56.617 INFO:teuthology.orchestra.run.smithi119.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:28:56.618 INFO:teuthology.orchestra.run.smithi119.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:28:56.629 INFO:teuthology.orchestra.run.smithi119.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T01:28:56.630 INFO:teuthology.orchestra.run.smithi119.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T01:28:56.630 INFO:teuthology.orchestra.run.smithi119.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T01:28:56.631 INFO:teuthology.orchestra.run.smithi119.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T01:28:56.633 INFO:teuthology.orchestra.run.smithi119.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T01:28:56.635 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T01:28:56.636 INFO:teuthology.orchestra.run.smithi119.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T01:28:57.126 INFO:teuthology.orchestra.run.smithi119.stdout:Dependencies resolved. 2023-06-23T01:28:57.140 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================================================== 2023-06-23T01:28:57.141 INFO:teuthology.orchestra.run.smithi119.stdout: Package Arch Version Repository Size 2023-06-23T01:28:57.141 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================================================== 2023-06-23T01:28:57.141 INFO:teuthology.orchestra.run.smithi119.stdout:Installing: 2023-06-23T01:28:57.141 INFO:teuthology.orchestra.run.smithi119.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T01:28:57.141 INFO:teuthology.orchestra.run.smithi119.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T01:28:57.141 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T01:28:57.142 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T01:28:57.142 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T01:28:57.142 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T01:28:57.142 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T01:28:57.142 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T01:28:57.142 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T01:28:57.142 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T01:28:57.143 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:28:57.143 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T01:28:57.143 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T01:28:57.143 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T01:28:57.143 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T01:28:57.143 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T01:28:57.143 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T01:28:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T01:28:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T01:28:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T01:28:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T01:28:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T01:28:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T01:28:57.144 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T01:28:57.145 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T01:28:57.145 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T01:28:57.145 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T01:28:57.145 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T01:28:57.145 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T01:28:57.145 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T01:28:57.145 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:28:57.145 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T01:28:57.146 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T01:28:57.146 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrading: 2023-06-23T01:28:57.146 INFO:teuthology.orchestra.run.smithi119.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:28:57.146 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T01:28:57.146 INFO:teuthology.orchestra.run.smithi119.stdout:Installing dependencies: 2023-06-23T01:28:57.146 INFO:teuthology.orchestra.run.smithi119.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T01:28:57.146 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T01:28:57.147 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T01:28:57.147 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T01:28:57.147 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T01:28:57.147 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T01:28:57.147 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:28:57.147 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T01:28:57.147 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T01:28:57.147 INFO:teuthology.orchestra.run.smithi119.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T01:28:57.148 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T01:28:57.148 INFO:teuthology.orchestra.run.smithi119.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T01:28:57.148 INFO:teuthology.orchestra.run.smithi119.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T01:28:57.148 INFO:teuthology.orchestra.run.smithi119.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T01:28:57.148 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T01:28:57.148 INFO:teuthology.orchestra.run.smithi119.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T01:28:57.148 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T01:28:57.149 INFO:teuthology.orchestra.run.smithi119.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T01:28:57.149 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T01:28:57.149 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T01:28:57.149 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T01:28:57.149 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T01:28:57.149 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T01:28:57.149 INFO:teuthology.orchestra.run.smithi119.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T01:28:57.149 INFO:teuthology.orchestra.run.smithi119.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T01:28:57.150 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T01:28:57.150 INFO:teuthology.orchestra.run.smithi119.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T01:28:57.150 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T01:28:57.150 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T01:28:57.151 INFO:teuthology.orchestra.run.smithi119.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T01:28:57.151 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T01:28:57.151 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T01:28:57.151 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T01:28:57.151 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T01:28:57.151 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T01:28:57.151 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T01:28:57.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T01:28:57.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T01:28:57.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T01:28:57.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T01:28:57.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T01:28:57.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T01:28:57.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T01:28:57.152 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T01:28:57.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T01:28:57.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T01:28:57.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T01:28:57.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T01:28:57.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T01:28:57.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T01:28:57.153 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:28:57.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T01:28:57.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T01:28:57.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T01:28:57.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T01:28:57.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T01:28:57.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T01:28:57.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T01:28:57.154 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T01:28:57.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T01:28:57.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T01:28:57.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T01:28:57.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T01:28:57.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T01:28:57.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T01:28:57.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T01:28:57.155 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T01:28:57.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:28:57.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T01:28:57.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T01:28:57.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T01:28:57.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T01:28:57.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T01:28:57.156 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T01:28:57.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T01:28:57.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T01:28:57.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T01:28:57.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T01:28:57.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T01:28:57.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T01:28:57.157 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T01:28:57.157 INFO:teuthology.orchestra.run.smithi119.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T01:28:57.158 INFO:teuthology.orchestra.run.smithi119.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T01:28:57.158 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T01:28:57.158 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T01:28:57.158 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:28:57.158 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction Summary 2023-06-23T01:28:57.158 INFO:teuthology.orchestra.run.smithi119.stdout:====================================================================================================================== 2023-06-23T01:28:57.158 INFO:teuthology.orchestra.run.smithi119.stdout:Install 118 Packages 2023-06-23T01:28:57.158 INFO:teuthology.orchestra.run.smithi119.stdout:Upgrade 2 Packages 2023-06-23T01:28:57.159 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:28:57.159 INFO:teuthology.orchestra.run.smithi119.stdout:Total download size: 201 M 2023-06-23T01:28:57.159 INFO:teuthology.orchestra.run.smithi119.stdout:Downloading Packages: 2023-06-23T01:28:57.637 INFO:teuthology.orchestra.run.smithi119.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 42 kB/s | 6.4 kB 00:00 2023-06-23T01:28:58.259 INFO:teuthology.orchestra.run.smithi119.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 1.4 MB/s | 873 kB 00:00 2023-06-23T01:28:58.436 INFO:teuthology.orchestra.run.smithi119.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 920 kB/s | 162 kB 00:00 2023-06-23T01:28:58.758 INFO:teuthology.orchestra.run.smithi119.stdout:(4/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 18 MB/s | 23 MB 00:01 2023-06-23T01:28:58.847 INFO:teuthology.orchestra.run.smithi119.stdout:(5/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 18 MB/s | 1.6 MB 00:00 2023-06-23T01:28:59.006 INFO:teuthology.orchestra.run.smithi119.stdout:(6/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 3.4 MB/s | 5.2 MB 00:01 2023-06-23T01:28:59.051 INFO:teuthology.orchestra.run.smithi119.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 19 MB/s | 3.9 MB 00:00 2023-06-23T01:28:59.164 INFO:teuthology.orchestra.run.smithi119.stdout:(8/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 3.2 MB/s | 2.3 MB 00:00 2023-06-23T01:28:59.236 INFO:teuthology.orchestra.run.smithi119.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 338 kB/s | 24 kB 00:00 2023-06-23T01:29:00.081 INFO:teuthology.orchestra.run.smithi119.stdout:(10/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 16 MB/s | 17 MB 00:01 2023-06-23T01:29:00.129 INFO:teuthology.orchestra.run.smithi119.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 5.3 MB/s | 255 kB 00:00 2023-06-23T01:29:00.170 INFO:teuthology.orchestra.run.smithi119.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 600 kB/s | 25 kB 00:00 2023-06-23T01:29:00.234 INFO:teuthology.orchestra.run.smithi119.stdout:(13/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 12 MB/s | 794 kB 00:00 2023-06-23T01:29:00.287 INFO:teuthology.orchestra.run.smithi092.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 122 MB/s | 38 MB 00:00 2023-06-23T01:29:00.288 INFO:teuthology.orchestra.run.smithi119.stdout:(14/120): libcephsqlite-17.2.6-719.g42ddc46e.el 3.2 MB/s | 174 kB 00:00 2023-06-23T01:29:00.336 INFO:teuthology.orchestra.run.smithi119.stdout:(15/120): librados-devel-17.2.6-719.g42ddc46e.e 2.6 MB/s | 130 kB 00:00 2023-06-23T01:29:00.400 INFO:teuthology.orchestra.run.smithi119.stdout:(16/120): libradosstriper1-17.2.6-719.g42ddc46e 7.9 MB/s | 506 kB 00:00 2023-06-23T01:29:00.775 INFO:teuthology.orchestra.run.smithi119.stdout:(17/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 9.4 MB/s | 17 MB 00:01 2023-06-23T01:29:00.819 INFO:teuthology.orchestra.run.smithi119.stdout:(18/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 13 MB/s | 5.5 MB 00:00 2023-06-23T01:29:00.822 INFO:teuthology.orchestra.run.smithi119.stdout:(19/120): python3-ceph-argparse-17.2.6-719.g42d 956 kB/s | 45 kB 00:00 2023-06-23T01:29:00.843 INFO:teuthology.orchestra.run.smithi119.stdout:(20/120): python3-ceph-common-17.2.6-719.g42ddc 4.9 MB/s | 115 kB 00:00 2023-06-23T01:29:00.866 INFO:teuthology.orchestra.run.smithi119.stdout:(21/120): python3-cephfs-17.2.6-719.g42ddc46e.e 4.3 MB/s | 190 kB 00:00 2023-06-23T01:29:00.867 INFO:teuthology.orchestra.run.smithi119.stdout:(22/120): python3-rados-17.2.6-719.g42ddc46e.el 15 MB/s | 371 kB 00:00 2023-06-23T01:29:00.897 INFO:teuthology.orchestra.run.smithi119.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 3.5 MB/s | 107 kB 00:00 2023-06-23T01:29:00.909 INFO:teuthology.orchestra.run.smithi119.stdout:(24/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 8.1 MB/s | 357 kB 00:00 2023-06-23T01:29:00.918 INFO:teuthology.orchestra.run.smithi119.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 4.2 MB/s | 91 kB 00:00 2023-06-23T01:29:00.956 INFO:teuthology.orchestra.run.smithi119.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 4.6 MB/s | 173 kB 00:00 2023-06-23T01:29:00.992 INFO:teuthology.orchestra.run.smithi119.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 636 kB/s | 23 kB 00:00 2023-06-23T01:29:01.027 INFO:teuthology.orchestra.run.smithi119.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 3.4 MB/s | 121 kB 00:00 2023-06-23T01:29:01.154 INFO:teuthology.orchestra.run.smithi119.stdout:(29/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 14 MB/s | 3.5 MB 00:00 2023-06-23T01:29:01.172 INFO:teuthology.orchestra.run.smithi119.stdout:(30/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 12 MB/s | 1.7 MB 00:00 2023-06-23T01:29:01.220 INFO:teuthology.orchestra.run.smithi119.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 4.7 MB/s | 238 kB 00:00 2023-06-23T01:29:01.260 INFO:teuthology.orchestra.run.smithi119.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 1.1 MB/s | 47 kB 00:00 2023-06-23T01:29:01.294 INFO:teuthology.orchestra.run.smithi119.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 431 kB/s | 14 kB 00:00 2023-06-23T01:29:01.336 INFO:teuthology.orchestra.run.smithi119.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 6.2 MB/s | 264 kB 00:00 2023-06-23T01:29:01.379 INFO:teuthology.orchestra.run.smithi119.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.8 MB/s | 80 kB 00:00 2023-06-23T01:29:01.415 INFO:teuthology.orchestra.run.smithi119.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 498 kB/s | 18 kB 00:00 2023-06-23T01:29:01.431 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:01.433 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:01.615 INFO:teuthology.orchestra.run.smithi119.stdout:(37/120): ceph-mgr-diskprediction-local-17.2.6- 16 MB/s | 7.4 MB 00:00 2023-06-23T01:29:01.616 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:01.654 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:01.743 INFO:teuthology.orchestra.run.smithi119.stdout:(38/120): gperftools-libs-2.7-9.el8.x86_64.rpm 936 kB/s | 306 kB 00:00 2023-06-23T01:29:01.744 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:01.745 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:01.798 INFO:teuthology.orchestra.run.smithi119.stdout:(39/120): libunwind-1.3.1-3.el8.x86_64.rpm 1.3 MB/s | 75 kB 00:00 2023-06-23T01:29:01.798 INFO:teuthology.orchestra.run.smithi119.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T01:29:01.863 INFO:teuthology.orchestra.run.smithi119.stdout:(40/120): liboath-2.6.2-3.el8.x86_64.rpm 237 kB/s | 59 kB 00:00 2023-06-23T01:29:01.904 INFO:teuthology.orchestra.run.smithi119.stdout:(41/120): python3-asyncssh-2.7.0-2.el8.noarch.r 4.4 MB/s | 476 kB 00:00 2023-06-23T01:29:01.917 INFO:teuthology.orchestra.run.smithi119.stdout:(42/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 808 kB/s | 44 kB 00:00 2023-06-23T01:29:01.958 INFO:teuthology.orchestra.run.smithi119.stdout:(43/120): python3-beautifulsoup4-4.6.3-2.el8.1. 3.4 MB/s | 185 kB 00:00 2023-06-23T01:29:01.970 INFO:teuthology.orchestra.run.smithi119.stdout:(44/120): python3-cachetools-3.1.1-4.el8.noarch 625 kB/s | 33 kB 00:00 2023-06-23T01:29:02.009 INFO:teuthology.orchestra.run.smithi119.stdout:(45/120): python3-certifi-2018.10.15-7.el8.noar 317 kB/s | 16 kB 00:00 2023-06-23T01:29:02.063 INFO:teuthology.orchestra.run.smithi119.stdout:(46/120): python3-cherrypy-18.4.0-1.el8.noarch. 7.0 MB/s | 384 kB 00:00 2023-06-23T01:29:02.079 INFO:teuthology.orchestra.run.smithi119.stdout:(47/120): python3-cheroot-8.5.2-1.el8.noarch.rp 1.5 MB/s | 173 kB 00:00 2023-06-23T01:29:02.113 INFO:teuthology.orchestra.run.smithi119.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 1.6 MB/s | 81 kB 00:00 2023-06-23T01:29:02.130 INFO:teuthology.orchestra.run.smithi119.stdout:(49/120): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-06-23T01:29:02.162 INFO:teuthology.orchestra.run.smithi119.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 358 kB/s | 18 kB 00:00 2023-06-23T01:29:02.212 INFO:teuthology.orchestra.run.smithi119.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 986 kB/s | 49 kB 00:00 2023-06-23T01:29:02.269 INFO:teuthology.orchestra.run.smithi119.stdout:(52/120): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-06-23T01:29:02.279 INFO:teuthology.orchestra.run.smithi119.stdout:(53/120): python3-kubernetes-11.0.0-6.el8.noarc 7.8 MB/s | 1.2 MB 00:00 2023-06-23T01:29:02.321 INFO:teuthology.orchestra.run.smithi119.stdout:(54/120): python3-natsort-7.1.1-2.el8.noarch.rp 1.1 MB/s | 60 kB 00:00 2023-06-23T01:29:02.330 INFO:teuthology.orchestra.run.smithi119.stdout:(55/120): python3-pecan-1.3.2-9.el8.noarch.rpm 5.5 MB/s | 283 kB 00:00 2023-06-23T01:29:02.372 INFO:teuthology.orchestra.run.smithi119.stdout:(56/120): python3-portend-2.6-1.el8.noarch.rpm 312 kB/s | 16 kB 00:00 2023-06-23T01:29:02.382 INFO:teuthology.orchestra.run.smithi119.stdout:(57/120): python3-repoze-lru-0.7-6.el8.noarch.r 646 kB/s | 34 kB 00:00 2023-06-23T01:29:02.421 INFO:teuthology.orchestra.run.smithi119.stdout:(58/120): python3-routes-2.4.1-12.el8.noarch.rp 3.9 MB/s | 196 kB 00:00 2023-06-23T01:29:02.436 INFO:teuthology.orchestra.run.smithi119.stdout:(59/120): python3-rsa-4.8-1.el8.noarch.rpm 1.1 MB/s | 61 kB 00:00 2023-06-23T01:29:02.467 INFO:teuthology.orchestra.run.smithi119.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 402 kB/s | 19 kB 00:00 2023-06-23T01:29:02.491 INFO:teuthology.orchestra.run.smithi119.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 438 kB/s | 24 kB 00:00 2023-06-23T01:29:02.516 INFO:teuthology.orchestra.run.smithi119.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 611 kB/s | 29 kB 00:00 2023-06-23T01:29:02.635 INFO:teuthology.orchestra.run.smithi119.stdout:(63/120): python3-trustme-0.6.0-4.el8.noarch.rp 191 kB/s | 27 kB 00:00 2023-06-23T01:29:02.638 INFO:teuthology.orchestra.run.smithi119.stdout:(64/120): python3-waitress-1.4.3-1.el8.noarch.r 2.0 MB/s | 255 kB 00:00 2023-06-23T01:29:02.689 INFO:teuthology.orchestra.run.smithi119.stdout:(65/120): python3-webob-1.8.5-1.el8.1.noarch.rp 4.6 MB/s | 251 kB 00:00 2023-06-23T01:29:02.694 INFO:teuthology.orchestra.run.smithi119.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 1.1 MB/s | 61 kB 00:00 2023-06-23T01:29:02.740 INFO:teuthology.orchestra.run.smithi119.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 1.6 MB/s | 85 kB 00:00 2023-06-23T01:29:02.751 INFO:teuthology.orchestra.run.smithi119.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 406 kB/s | 23 kB 00:00 2023-06-23T01:29:02.755 INFO:teuthology.orchestra.run.smithi119.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 16 MB/s | 70 kB 00:00 2023-06-23T01:29:02.799 INFO:teuthology.orchestra.run.smithi119.stdout:(70/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 427 kB/s | 19 kB 00:00 2023-06-23T01:29:02.821 INFO:teuthology.orchestra.run.smithi119.stdout:(71/120): python3-jwt-1.6.1-2.el8.noarch.rpm 1.9 MB/s | 43 kB 00:00 2023-06-23T01:29:02.845 INFO:teuthology.orchestra.run.smithi119.stdout:(72/120): thrift-0.13.0-2.el8.x86_64.rpm 17 MB/s | 1.7 MB 00:00 2023-06-23T01:29:02.848 INFO:teuthology.orchestra.run.smithi119.stdout:(73/120): python3-oauthlib-2.1.0-1.el8.noarch.r 5.7 MB/s | 155 kB 00:00 2023-06-23T01:29:02.880 INFO:teuthology.orchestra.run.smithi119.stdout:(74/120): keyutils-libs-devel-1.5.10-6.el8.x86_ 1.5 MB/s | 48 kB 00:00 2023-06-23T01:29:02.887 INFO:teuthology.orchestra.run.smithi119.stdout:(75/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 5.2 MB/s | 220 kB 00:00 2023-06-23T01:29:02.912 INFO:teuthology.orchestra.run.smithi119.stdout:(76/120): python3-requests-oauthlib-1.0.0-1.el8 1.7 MB/s | 43 kB 00:00 2023-06-23T01:29:02.914 INFO:teuthology.orchestra.run.smithi119.stdout:(77/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 6.6 MB/s | 229 kB 00:00 2023-06-23T01:29:02.942 INFO:teuthology.orchestra.run.smithi119.stdout:(78/120): libverto-devel-0.3.0-5.el8.x86_64.rpm 643 kB/s | 18 kB 00:00 2023-06-23T01:29:02.962 INFO:teuthology.orchestra.run.smithi119.stdout:(79/120): openssl-devel-1.1.1g-15.el8_3.x86_64. 46 MB/s | 2.3 MB 00:00 2023-06-23T01:29:02.972 INFO:teuthology.orchestra.run.smithi119.stdout:(80/120): pcre2-devel-10.32-2.el8.x86_64.rpm 20 MB/s | 605 kB 00:00 2023-06-23T01:29:02.993 INFO:teuthology.orchestra.run.smithi119.stdout:(81/120): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 83 kB 00:00 2023-06-23T01:29:02.998 INFO:teuthology.orchestra.run.smithi119.stdout:(82/120): elfutils-libelf-devel-0.182-3.el8.x86 2.2 MB/s | 59 kB 00:00 2023-06-23T01:29:03.023 INFO:teuthology.orchestra.run.smithi119.stdout:(83/120): libcom_err-devel-1.45.6-1.el8.x86_64. 1.5 MB/s | 38 kB 00:00 2023-06-23T01:29:03.027 INFO:teuthology.orchestra.run.smithi119.stdout:(84/120): libquadmath-8.4.1-1.el8.x86_64.rpm 4.8 MB/s | 169 kB 00:00 2023-06-23T01:29:03.074 INFO:teuthology.orchestra.run.smithi119.stdout:(85/120): libsepol-devel-2.9-2.el8.x86_64.rpm 1.6 MB/s | 86 kB 00:00 2023-06-23T01:29:03.075 INFO:teuthology.orchestra.run.smithi119.stdout:(86/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 1.2 MB/s | 58 kB 00:00 2023-06-23T01:29:03.101 INFO:teuthology.orchestra.run.smithi119.stdout:(87/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.7 MB/s | 47 kB 00:00 2023-06-23T01:29:03.116 INFO:teuthology.orchestra.run.smithi119.stdout:(88/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.rp 16 MB/s | 559 kB 00:00 2023-06-23T01:29:03.132 INFO:teuthology.orchestra.run.smithi119.stdout:(89/120): libgfortran-8.4.1-1.el8.x86_64.rpm 20 MB/s | 639 kB 00:00 2023-06-23T01:29:03.135 INFO:teuthology.orchestra.run.smithi119.stdout:(90/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rpm 2.7 MB/s | 70 kB 00:00 2023-06-23T01:29:03.159 INFO:teuthology.orchestra.run.smithi119.stdout:(91/120): libselinux-devel-2.9-5.el8.x86_64.rpm 7.2 MB/s | 200 kB 00:00 2023-06-23T01:29:03.165 INFO:teuthology.orchestra.run.smithi119.stdout:(92/120): libuuid-devel-2.32.1-27.el8.x86_64.rp 3.3 MB/s | 97 kB 00:00 2023-06-23T01:29:03.181 INFO:teuthology.orchestra.run.smithi119.stdout:(93/120): python3-prettytable-0.7.2-14.el8.noar 2.6 MB/s | 44 kB 00:00 2023-06-23T01:29:03.196 INFO:teuthology.orchestra.run.smithi119.stdout:(94/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.8 MB/s | 27 kB 00:00 2023-06-23T01:29:03.211 INFO:teuthology.orchestra.run.smithi119.stdout:(95/120): jq-1.5-12.el8.x86_64.rpm 10 MB/s | 161 kB 00:00 2023-06-23T01:29:03.229 INFO:teuthology.orchestra.run.smithi119.stdout:(96/120): flex-2.6.1-9.el8.x86_64.rpm 18 MB/s | 320 kB 00:00 2023-06-23T01:29:03.249 INFO:teuthology.orchestra.run.smithi119.stdout:(97/120): python3-werkzeug-0.12.2-4.el8.noarch. 23 MB/s | 457 kB 00:00 2023-06-23T01:29:03.416 INFO:teuthology.orchestra.run.smithi119.stdout:(98/120): python3-scipy-1.0.0-20.module+el8.1.0 57 MB/s | 15 MB 00:00 2023-06-23T01:29:03.431 INFO:teuthology.orchestra.run.smithi119.stdout:(99/120): python3-babel-2.5.1-5.el8.noarch.rpm 26 MB/s | 4.8 MB 00:00 2023-06-23T01:29:03.432 INFO:teuthology.orchestra.run.smithi119.stdout:(100/120): python3-markupsafe-0.23-19.el8.x86_6 2.4 MB/s | 39 kB 00:00 2023-06-23T01:29:03.451 INFO:teuthology.orchestra.run.smithi119.stdout:(101/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 14 MB/s | 259 kB 00:00 2023-06-23T01:29:03.456 INFO:teuthology.orchestra.run.smithi119.stdout:(102/120): bison-3.0.4-10.el8.x86_64.rpm 27 MB/s | 688 kB 00:00 2023-06-23T01:29:03.467 INFO:teuthology.orchestra.run.smithi119.stdout:(103/120): python3-numpy-f2py-1.14.3-9.el8.x86_ 14 MB/s | 225 kB 00:00 2023-06-23T01:29:03.474 INFO:teuthology.orchestra.run.smithi119.stdout:(104/120): python3-pyasn1-0.3.7-6.el8.noarch.rp 7.0 MB/s | 126 kB 00:00 2023-06-23T01:29:03.492 INFO:teuthology.orchestra.run.smithi119.stdout:(105/120): oniguruma-6.8.2-2.el8.x86_64.rpm 10 MB/s | 187 kB 00:00 2023-06-23T01:29:03.511 INFO:teuthology.orchestra.run.smithi119.stdout:(106/120): python3-jinja2-2.10.1-2.el8_0.noarch 29 MB/s | 538 kB 00:00 2023-06-23T01:29:03.523 INFO:teuthology.orchestra.run.smithi119.stdout:(107/120): python3-numpy-1.14.3-9.el8.x86_64.rp 67 MB/s | 3.7 MB 00:00 2023-06-23T01:29:03.527 INFO:teuthology.orchestra.run.smithi119.stdout:(108/120): socat-1.7.3.3-2.el8.x86_64.rpm 19 MB/s | 302 kB 00:00 2023-06-23T01:29:03.538 INFO:teuthology.orchestra.run.smithi119.stdout:(109/120): python3-mako-1.0.6-13.el8.noarch.rpm 10 MB/s | 157 kB 00:00 2023-06-23T01:29:03.544 INFO:teuthology.orchestra.run.smithi119.stdout:(110/120): python3-pyasn1-modules-0.3.7-6.el8.n 6.1 MB/s | 110 kB 00:00 2023-06-23T01:29:03.554 INFO:teuthology.orchestra.run.smithi119.stdout:(111/120): librdkafka-0.11.4-1.el8.x86_64.rpm 22 MB/s | 353 kB 00:00 2023-06-23T01:29:03.561 INFO:teuthology.orchestra.run.smithi119.stdout:(112/120): python3-jsonpointer-1.10-11.el8.noar 1.2 MB/s | 20 kB 00:00 2023-06-23T01:29:03.576 INFO:teuthology.orchestra.run.smithi119.stdout:(113/120): libtool-2.4.6-25.el8.x86_64.rpm 31 MB/s | 709 kB 00:00 2023-06-23T01:29:03.577 INFO:teuthology.orchestra.run.smithi119.stdout:(114/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.4 MB/s | 54 kB 00:00 2023-06-23T01:29:03.652 INFO:teuthology.orchestra.run.smithi119.stdout:(115/120): openblas-0.3.12-1.el8.x86_64.rpm 61 MB/s | 4.6 MB 00:00 2023-06-23T01:29:03.666 INFO:teuthology.orchestra.run.smithi119.stdout:(116/120): openblas-threads-0.3.12-1.el8.x86_64 53 MB/s | 4.7 MB 00:00 2023-06-23T01:29:03.674 INFO:teuthology.orchestra.run.smithi119.stdout:(117/120): automake-1.16.1-7.el8.noarch.rpm 33 MB/s | 713 kB 00:00 2023-06-23T01:29:04.376 INFO:teuthology.orchestra.run.smithi119.stdout:(118/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 5.1 MB/s | 3.6 MB 00:00 2023-06-23T01:29:04.848 INFO:teuthology.orchestra.run.smithi119.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 2.9 MB/s | 3.5 MB 00:01 2023-06-23T01:29:06.833 INFO:teuthology.orchestra.run.smithi119.stdout:(120/120): ceph-test-17.2.6-719.g42ddc46e.el8.x 6.8 MB/s | 52 MB 00:07 2023-06-23T01:29:06.838 INFO:teuthology.orchestra.run.smithi119.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:29:06.839 INFO:teuthology.orchestra.run.smithi119.stdout:Total 21 MB/s | 201 MB 00:09 2023-06-23T01:29:06.891 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction check 2023-06-23T01:29:07.007 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction check succeeded. 2023-06-23T01:29:07.008 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction test 2023-06-23T01:29:08.738 INFO:teuthology.orchestra.run.smithi092.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:29:08.739 INFO:teuthology.orchestra.run.smithi092.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-06-23T01:29:08.750 INFO:teuthology.orchestra.run.smithi092.stdout:Package NetworkManager-1:1.30.0-13.el8_4.x86_64 is already installed. 2023-06-23T01:29:08.751 INFO:teuthology.orchestra.run.smithi092.stdout:Package iproute-5.9.0-4.el8.x86_64 is already installed. 2023-06-23T01:29:08.751 INFO:teuthology.orchestra.run.smithi092.stdout:Package util-linux-2.32.1-27.el8.x86_64 is already installed. 2023-06-23T01:29:08.752 INFO:teuthology.orchestra.run.smithi092.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-06-23T01:29:08.754 INFO:teuthology.orchestra.run.smithi092.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-06-23T01:29:08.756 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsdump-3.1.8-2.el8.x86_64 is already installed. 2023-06-23T01:29:08.757 INFO:teuthology.orchestra.run.smithi092.stdout:Package xfsprogs-5.0.0-8.el8.x86_64 is already installed. 2023-06-23T01:29:08.766 INFO:teuthology.orchestra.run.smithi119.stdout:Transaction test succeeded. 2023-06-23T01:29:08.770 INFO:teuthology.orchestra.run.smithi119.stdout:Running transaction 2023-06-23T01:29:09.245 INFO:teuthology.orchestra.run.smithi092.stdout:Dependencies resolved. 2023-06-23T01:29:09.256 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================================================== 2023-06-23T01:29:09.256 INFO:teuthology.orchestra.run.smithi092.stdout: Package Arch Version Repository Size 2023-06-23T01:29:09.256 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================================================== 2023-06-23T01:29:09.256 INFO:teuthology.orchestra.run.smithi092.stdout:Installing: 2023-06-23T01:29:09.256 INFO:teuthology.orchestra.run.smithi092.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-06-23T01:29:09.257 INFO:teuthology.orchestra.run.smithi092.stdout: ceph x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 6.4 k 2023-06-23T01:29:09.257 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.2 M 2023-06-23T01:29:09.257 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 873 k 2023-06-23T01:29:09.257 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 162 k 2023-06-23T01:29:09.257 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 1.6 M 2023-06-23T01:29:09.257 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 121 k 2023-06-23T01:29:09.257 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 1.7 M 2023-06-23T01:29:09.258 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 7.4 M 2023-06-23T01:29:09.258 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 47 k 2023-06-23T01:29:09.258 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:29:09.258 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 52 M 2023-06-23T01:29:09.258 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 264 k 2023-06-23T01:29:09.258 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 80 k 2023-06-23T01:29:09.258 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 255 k 2023-06-23T01:29:09.258 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 18 k 2023-06-23T01:29:09.259 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel x86_64 0.182-3.el8 rhel-8-for-x86_64-baseos-rpms 59 k 2023-06-23T01:29:09.259 INFO:teuthology.orchestra.run.smithi092.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-06-23T01:29:09.259 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-06-23T01:29:09.259 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 25 k 2023-06-23T01:29:09.259 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 794 k 2023-06-23T01:29:09.259 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 130 k 2023-06-23T01:29:09.259 INFO:teuthology.orchestra.run.smithi092.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-06-23T01:29:09.260 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel x86_64 2.32.1-27.el8 rhel-8-for-x86_64-baseos-rpms 97 k 2023-06-23T01:29:09.260 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel x86_64 1:1.1.1g-15.el8_3 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-06-23T01:29:09.260 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 190 k 2023-06-23T01:29:09.260 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 371 k 2023-06-23T01:29:09.260 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 357 k 2023-06-23T01:29:09.260 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 107 k 2023-06-23T01:29:09.260 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 91 k 2023-06-23T01:29:09.260 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:29:09.261 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 173 k 2023-06-23T01:29:09.261 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel x86_64 5.0.0-8.el8 rhel-8-for-x86_64-baseos-rpms 70 k 2023-06-23T01:29:09.261 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrading: 2023-06-23T01:29:09.261 INFO:teuthology.orchestra.run.smithi092.stdout: librados2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.5 M 2023-06-23T01:29:09.261 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.6 M 2023-06-23T01:29:09.261 INFO:teuthology.orchestra.run.smithi092.stdout:Installing dependencies: 2023-06-23T01:29:09.261 INFO:teuthology.orchestra.run.smithi092.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-06-23T01:29:09.262 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 23 M 2023-06-23T01:29:09.262 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 23 k 2023-06-23T01:29:09.262 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 2.3 M 2023-06-23T01:29:09.262 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 238 k 2023-06-23T01:29:09.262 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 3.9 M 2023-06-23T01:29:09.262 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 17 M 2023-06-23T01:29:09.262 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts noarch 2:17.2.6-719.g42ddc46e.el8 ceph-noarch 14 k 2023-06-23T01:29:09.262 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 24 k 2023-06-23T01:29:09.263 INFO:teuthology.orchestra.run.smithi092.stdout: fuse x86_64 2.9.7-12.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-06-23T01:29:09.263 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-06-23T01:29:09.263 INFO:teuthology.orchestra.run.smithi092.stdout: jq x86_64 1.5-12.el8 rhel-8-for-x86_64-appstream-rpms 161 k 2023-06-23T01:29:09.263 INFO:teuthology.orchestra.run.smithi092.stdout: keyutils-libs-devel x86_64 1.5.10-6.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-06-23T01:29:09.263 INFO:teuthology.orchestra.run.smithi092.stdout: krb5-devel x86_64 1.18.2-8.3.el8_4 rhel-8-for-x86_64-baseos-rpms 559 k 2023-06-23T01:29:09.263 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 174 k 2023-06-23T01:29:09.263 INFO:teuthology.orchestra.run.smithi092.stdout: libcom_err-devel x86_64 1.45.6-1.el8 rhel-8-for-x86_64-baseos-rpms 38 k 2023-06-23T01:29:09.263 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 639 k 2023-06-23T01:29:09.264 INFO:teuthology.orchestra.run.smithi092.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-06-23T01:29:09.264 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath x86_64 8.4.1-1.el8 rhel-8-for-x86_64-baseos-rpms 169 k 2023-06-23T01:29:09.264 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-06-23T01:29:09.264 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 506 k 2023-06-23T01:29:09.264 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka x86_64 0.11.4-1.el8 rhel-8-for-x86_64-appstream-rpms 353 k 2023-06-23T01:29:09.264 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2 x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 5.5 M 2023-06-23T01:29:09.264 INFO:teuthology.orchestra.run.smithi092.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-06-23T01:29:09.264 INFO:teuthology.orchestra.run.smithi092.stdout: libsepol-devel x86_64 2.9-2.el8 rhel-8-for-x86_64-baseos-rpms 86 k 2023-06-23T01:29:09.265 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-06-23T01:29:09.265 INFO:teuthology.orchestra.run.smithi092.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-06-23T01:29:09.265 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-06-23T01:29:09.265 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-06-23T01:29:09.265 INFO:teuthology.orchestra.run.smithi092.stdout: openblas x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.6 M 2023-06-23T01:29:09.265 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-threads x86_64 0.3.12-1.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-06-23T01:29:09.265 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-06-23T01:29:09.266 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-06-23T01:29:09.266 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-06-23T01:29:09.266 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-06-23T01:29:09.266 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel noarch 2.5.1-5.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-06-23T01:29:09.266 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-06-23T01:29:09.266 INFO:teuthology.orchestra.run.smithi092.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-06-23T01:29:09.266 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-06-23T01:29:09.266 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 45 k 2023-06-23T01:29:09.267 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common x86_64 2:17.2.6-719.g42ddc46e.el8 ceph 115 k 2023-06-23T01:29:09.267 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-06-23T01:29:09.267 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-06-23T01:29:09.267 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-06-23T01:29:09.267 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-06-23T01:29:09.267 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-06-23T01:29:09.267 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-06-23T01:29:09.267 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 rhel-8-for-x86_64-appstream-rpms 538 k 2023-06-23T01:29:09.268 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-06-23T01:29:09.268 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-06-23T01:29:09.268 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:29:09.268 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-06-23T01:29:09.268 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-06-23T01:29:09.268 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-06-23T01:29:09.268 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-06-23T01:29:09.269 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-06-23T01:29:09.269 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-06-23T01:29:09.269 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-06-23T01:29:09.269 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-06-23T01:29:09.269 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-06-23T01:29:09.269 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-06-23T01:29:09.269 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-06-23T01:29:09.269 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-06-23T01:29:09.270 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-06-23T01:29:09.270 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-06-23T01:29:09.270 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-06-23T01:29:09.270 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-06-23T01:29:09.270 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-06-23T01:29:09.270 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-06-23T01:29:09.270 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-06-23T01:29:09.270 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy x86_64 1.0.0-20.module+el8.1.0+3334+5cb623d7 rhel-8-for-x86_64-appstream-rpms 15 M 2023-06-23T01:29:09.271 INFO:teuthology.orchestra.run.smithi092.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-06-23T01:29:09.271 INFO:teuthology.orchestra.run.smithi092.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-06-23T01:29:09.271 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-06-23T01:29:09.271 INFO:teuthology.orchestra.run.smithi092.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-06-23T01:29:09.271 INFO:teuthology.orchestra.run.smithi092.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-06-23T01:29:09.271 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-06-23T01:29:09.271 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-06-23T01:29:09.272 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-06-23T01:29:09.272 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-06-23T01:29:09.272 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-06-23T01:29:09.272 INFO:teuthology.orchestra.run.smithi092.stdout: socat x86_64 1.7.3.3-2.el8 rhel-8-for-x86_64-appstream-rpms 302 k 2023-06-23T01:29:09.272 INFO:teuthology.orchestra.run.smithi092.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-06-23T01:29:09.272 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-06-23T01:29:09.272 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel x86_64 1.2.11-17.el8 rhel-8-for-x86_64-baseos-rpms 58 k 2023-06-23T01:29:09.273 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:29:09.273 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction Summary 2023-06-23T01:29:09.273 INFO:teuthology.orchestra.run.smithi092.stdout:====================================================================================================================== 2023-06-23T01:29:09.273 INFO:teuthology.orchestra.run.smithi092.stdout:Install 118 Packages 2023-06-23T01:29:09.273 INFO:teuthology.orchestra.run.smithi092.stdout:Upgrade 2 Packages 2023-06-23T01:29:09.273 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:29:09.274 INFO:teuthology.orchestra.run.smithi092.stdout:Total download size: 201 M 2023-06-23T01:29:09.274 INFO:teuthology.orchestra.run.smithi092.stdout:Downloading Packages: 2023-06-23T01:29:09.579 INFO:teuthology.orchestra.run.smithi092.stdout:(1/120): ceph-17.2.6-719.g42ddc46e.el8.x86_64.r 43 kB/s | 6.4 kB 00:00 2023-06-23T01:29:09.863 INFO:teuthology.orchestra.run.smithi092.stdout:(2/120): ceph-fuse-17.2.6-719.g42ddc46e.el8.x86 3.0 MB/s | 873 kB 00:00 2023-06-23T01:29:09.922 INFO:teuthology.orchestra.run.smithi092.stdout:(3/120): ceph-immutable-object-cache-17.2.6-719 2.7 MB/s | 162 kB 00:00 2023-06-23T01:29:10.223 INFO:teuthology.orchestra.run.smithi092.stdout:(4/120): ceph-base-17.2.6-719.g42ddc46e.el8.x86 6.5 MB/s | 5.2 MB 00:00 2023-06-23T01:29:10.260 INFO:teuthology.orchestra.run.smithi092.stdout:(5/120): ceph-mds-17.2.6-719.g42ddc46e.el8.x86_ 6.8 MB/s | 2.3 MB 00:00 2023-06-23T01:29:10.623 INFO:teuthology.orchestra.run.smithi092.stdout:(6/120): ceph-mgr-17.2.6-719.g42ddc46e.el8.x86_ 3.9 MB/s | 1.6 MB 00:00 2023-06-23T01:29:10.713 INFO:teuthology.orchestra.run.smithi092.stdout:(7/120): ceph-mon-17.2.6-719.g42ddc46e.el8.x86_ 8.7 MB/s | 3.9 MB 00:00 2023-06-23T01:29:12.201 INFO:teuthology.orchestra.run.smithi092.stdout:(8/120): ceph-common-17.2.6-719.g42ddc46e.el8.x 8.4 MB/s | 23 MB 00:02 2023-06-23T01:29:12.266 INFO:teuthology.orchestra.run.smithi092.stdout:(9/120): ceph-selinux-17.2.6-719.g42ddc46e.el8. 376 kB/s | 24 kB 00:00 2023-06-23T01:29:12.506 INFO:teuthology.orchestra.run.smithi092.stdout:(10/120): ceph-osd-17.2.6-719.g42ddc46e.el8.x86 8.9 MB/s | 17 MB 00:01 2023-06-23T01:29:12.573 INFO:teuthology.orchestra.run.smithi092.stdout:(11/120): cephfs-mirror-17.2.6-719.g42ddc46e.el 3.7 MB/s | 255 kB 00:00 2023-06-23T01:29:12.632 INFO:teuthology.orchestra.run.smithi092.stdout:(12/120): libcephfs-devel-17.2.6-719.g42ddc46e. 428 kB/s | 25 kB 00:00 2023-06-23T01:29:12.805 INFO:teuthology.orchestra.run.smithi092.stdout:(13/120): ceph-radosgw-17.2.6-719.g42ddc46e.el8 8.0 MB/s | 17 MB 00:02 2023-06-23T01:29:12.810 INFO:teuthology.orchestra.run.smithi092.stdout:(14/120): libcephfs2-17.2.6-719.g42ddc46e.el8.x 4.4 MB/s | 794 kB 00:00 2023-06-23T01:29:12.864 INFO:teuthology.orchestra.run.smithi092.stdout:(15/120): libcephsqlite-17.2.6-719.g42ddc46e.el 2.9 MB/s | 174 kB 00:00 2023-06-23T01:29:12.868 INFO:teuthology.orchestra.run.smithi092.stdout:(16/120): librados-devel-17.2.6-719.g42ddc46e.e 2.2 MB/s | 130 kB 00:00 2023-06-23T01:29:12.920 INFO:teuthology.orchestra.run.smithi092.stdout:(17/120): libradosstriper1-17.2.6-719.g42ddc46e 8.9 MB/s | 506 kB 00:00 2023-06-23T01:29:12.983 INFO:teuthology.orchestra.run.smithi092.stdout:(18/120): python3-ceph-argparse-17.2.6-719.g42d 706 kB/s | 45 kB 00:00 2023-06-23T01:29:13.051 INFO:teuthology.orchestra.run.smithi092.stdout:(19/120): python3-ceph-common-17.2.6-719.g42ddc 1.7 MB/s | 115 kB 00:00 2023-06-23T01:29:13.116 INFO:teuthology.orchestra.run.smithi092.stdout:(20/120): python3-cephfs-17.2.6-719.g42ddc46e.e 2.9 MB/s | 190 kB 00:00 2023-06-23T01:29:13.196 INFO:teuthology.orchestra.run.smithi092.stdout:(21/120): python3-rados-17.2.6-719.g42ddc46e.el 4.5 MB/s | 371 kB 00:00 2023-06-23T01:29:13.269 INFO:teuthology.orchestra.run.smithi092.stdout:(22/120): python3-rbd-17.2.6-719.g42ddc46e.el8. 4.8 MB/s | 357 kB 00:00 2023-06-23T01:29:13.337 INFO:teuthology.orchestra.run.smithi092.stdout:(23/120): python3-rgw-17.2.6-719.g42ddc46e.el8. 1.5 MB/s | 107 kB 00:00 2023-06-23T01:29:13.371 INFO:teuthology.orchestra.run.smithi092.stdout:(24/120): librgw2-17.2.6-719.g42ddc46e.el8.x86_ 11 MB/s | 5.5 MB 00:00 2023-06-23T01:29:13.395 INFO:teuthology.orchestra.run.smithi092.stdout:(25/120): rbd-fuse-17.2.6-719.g42ddc46e.el8.x86 1.6 MB/s | 91 kB 00:00 2023-06-23T01:29:13.467 INFO:teuthology.orchestra.run.smithi092.stdout:(26/120): rbd-nbd-17.2.6-719.g42ddc46e.el8.x86_ 2.3 MB/s | 173 kB 00:00 2023-06-23T01:29:13.532 INFO:teuthology.orchestra.run.smithi092.stdout:(27/120): ceph-grafana-dashboards-17.2.6-719.g4 355 kB/s | 23 kB 00:00 2023-06-23T01:29:13.605 INFO:teuthology.orchestra.run.smithi092.stdout:(28/120): ceph-mgr-cephadm-17.2.6-719.g42ddc46e 1.6 MB/s | 121 kB 00:00 2023-06-23T01:29:13.717 INFO:teuthology.orchestra.run.smithi092.stdout:(29/120): rbd-mirror-17.2.6-719.g42ddc46e.el8.x 10 MB/s | 3.5 MB 00:00 2023-06-23T01:29:13.729 INFO:teuthology.orchestra.run.smithi119.stdout: Preparing : 1/1 2023-06-23T01:29:13.824 INFO:teuthology.orchestra.run.smithi092.stdout:(30/120): ceph-mgr-dashboard-17.2.6-719.g42ddc4 7.8 MB/s | 1.7 MB 00:00 2023-06-23T01:29:13.894 INFO:teuthology.orchestra.run.smithi092.stdout:(31/120): ceph-mgr-modules-core-17.2.6-719.g42d 3.3 MB/s | 238 kB 00:00 2023-06-23T01:29:13.956 INFO:teuthology.orchestra.run.smithi092.stdout:(32/120): ceph-mgr-rook-17.2.6-719.g42ddc46e.el 767 kB/s | 47 kB 00:00 2023-06-23T01:29:14.007 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T01:29:14.018 INFO:teuthology.orchestra.run.smithi092.stdout:(33/120): ceph-prometheus-alerts-17.2.6-719.g42 235 kB/s | 14 kB 00:00 2023-06-23T01:29:14.089 INFO:teuthology.orchestra.run.smithi092.stdout:(34/120): ceph-volume-17.2.6-719.g42ddc46e.el8. 3.6 MB/s | 264 kB 00:00 2023-06-23T01:29:14.154 INFO:teuthology.orchestra.run.smithi092.stdout:(35/120): cephadm-17.2.6-719.g42ddc46e.el8.noar 1.2 MB/s | 80 kB 00:00 2023-06-23T01:29:14.215 INFO:teuthology.orchestra.run.smithi092.stdout:(36/120): cephfs-top-17.2.6-719.g42ddc46e.el8.n 294 kB/s | 18 kB 00:00 2023-06-23T01:29:14.227 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T01:29:14.231 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:14.232 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] gperftools-libs-2.7-9.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/g/gperftools-libs-2.7-9.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:14.323 INFO:teuthology.orchestra.run.smithi092.stdout:(37/120): ceph-mgr-diskprediction-local-17.2.6- 12 MB/s | 7.4 MB 00:00 2023-06-23T01:29:14.329 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:14.350 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 2/122 2023-06-23T01:29:14.366 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] liboath-2.6.2-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/liboath-2.6.2-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:14.552 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T01:29:14.761 INFO:teuthology.orchestra.run.smithi092.stdout:(38/120): liboath-2.6.2-3.el8.x86_64.rpm 136 kB/s | 59 kB 00:00 2023-06-23T01:29:14.762 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:14.764 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] libunwind-1.3.1-3.el8.x86_64.rpm: Status code: 404 for http://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/l/libunwind-1.3.1-3.el8.x86_64.rpm (IP: 8.43.85.72) 2023-06-23T01:29:14.823 INFO:teuthology.orchestra.run.smithi092.stdout:(39/120): gperftools-libs-2.7-9.el8.x86_64.rpm 504 kB/s | 306 kB 00:00 2023-06-23T01:29:14.823 INFO:teuthology.orchestra.run.smithi092.stdout:[MIRROR] python3-asyncssh-2.7.0-2.el8.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/epel/8/Everything/x86_64/Packages/p/python3-asyncssh-2.7.0-2.el8.noarch.rpm (IP: 8.43.85.72) 2023-06-23T01:29:14.890 INFO:teuthology.orchestra.run.smithi092.stdout:(40/120): libunwind-1.3.1-3.el8.x86_64.rpm 587 kB/s | 75 kB 00:00 2023-06-23T01:29:14.971 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/122 2023-06-23T01:29:14.994 INFO:teuthology.orchestra.run.smithi092.stdout:(41/120): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 419 kB/s | 44 kB 00:00 2023-06-23T01:29:15.010 INFO:teuthology.orchestra.run.smithi092.stdout:(42/120): python3-asyncssh-2.7.0-2.el8.noarch.r 2.5 MB/s | 476 kB 00:00 2023-06-23T01:29:15.091 INFO:teuthology.orchestra.run.smithi092.stdout:(43/120): python3-cachetools-3.1.1-4.el8.noarch 411 kB/s | 33 kB 00:00 2023-06-23T01:29:15.171 INFO:teuthology.orchestra.run.smithi092.stdout:(44/120): python3-certifi-2018.10.15-7.el8.noar 205 kB/s | 16 kB 00:00 2023-06-23T01:29:15.178 INFO:teuthology.orchestra.run.smithi092.stdout:(45/120): python3-beautifulsoup4-4.6.3-2.el8.1. 1.0 MB/s | 185 kB 00:00 2023-06-23T01:29:15.201 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:29:15.261 INFO:teuthology.orchestra.run.smithi092.stdout:(46/120): python3-cheroot-8.5.2-1.el8.noarch.rp 1.9 MB/s | 173 kB 00:00 2023-06-23T01:29:15.594 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:29:15.747 INFO:teuthology.orchestra.run.smithi092.stdout:(47/120): ceph-test-17.2.6-719.g42ddc46e.el8.x8 15 MB/s | 52 MB 00:03 2023-06-23T01:29:15.748 INFO:teuthology.orchestra.run.smithi092.stdout:(48/120): python3-google-auth-1.1.1-10.el8.noar 167 kB/s | 81 kB 00:00 2023-06-23T01:29:15.749 INFO:teuthology.orchestra.run.smithi092.stdout:(49/120): python3-cherrypy-18.4.0-1.el8.noarch. 674 kB/s | 384 kB 00:00 2023-06-23T01:29:15.829 INFO:teuthology.orchestra.run.smithi092.stdout:(50/120): python3-jaraco-functools-2.0-4.el8.no 217 kB/s | 18 kB 00:00 2023-06-23T01:29:15.909 INFO:teuthology.orchestra.run.smithi092.stdout:(51/120): python3-logutils-0.3.5-11.el8.noarch. 608 kB/s | 49 kB 00:00 2023-06-23T01:29:15.967 INFO:teuthology.orchestra.run.smithi092.stdout:(52/120): python3-kubernetes-11.0.0-6.el8.noarc 5.4 MB/s | 1.2 MB 00:00 2023-06-23T01:29:15.990 INFO:teuthology.orchestra.run.smithi092.stdout:(53/120): python3-more-itertools-7.2.0-3.el8.no 730 kB/s | 59 kB 00:00 2023-06-23T01:29:16.000 INFO:teuthology.orchestra.run.smithi092.stdout:(54/120): python3-jaraco-6.2-6.el8.noarch.rpm 44 kB/s | 11 kB 00:00 2023-06-23T01:29:16.047 INFO:teuthology.orchestra.run.smithi092.stdout:(55/120): python3-natsort-7.1.1-2.el8.noarch.rp 762 kB/s | 60 kB 00:00 2023-06-23T01:29:16.090 INFO:teuthology.orchestra.run.smithi092.stdout:(56/120): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-06-23T01:29:16.101 INFO:teuthology.orchestra.run.smithi092.stdout:(57/120): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-06-23T01:29:16.124 INFO:teuthology.orchestra.run.smithi092.stdout:(58/120): python3-repoze-lru-0.7-6.el8.noarch.r 436 kB/s | 34 kB 00:00 2023-06-23T01:29:16.184 INFO:teuthology.orchestra.run.smithi092.stdout:(59/120): python3-routes-2.4.1-12.el8.noarch.rp 2.0 MB/s | 196 kB 00:00 2023-06-23T01:29:16.201 INFO:teuthology.orchestra.run.smithi092.stdout:(60/120): python3-simplegeneric-0.8.1-17.el8.no 244 kB/s | 19 kB 00:00 2023-06-23T01:29:16.243 INFO:teuthology.orchestra.run.smithi119.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:29:16.263 INFO:teuthology.orchestra.run.smithi092.stdout:(61/120): python3-singledispatch-3.4.0.3-18.el8 305 kB/s | 24 kB 00:00 2023-06-23T01:29:16.278 INFO:teuthology.orchestra.run.smithi092.stdout:(62/120): python3-tempora-1.14.1-5.el8.noarch.r 379 kB/s | 29 kB 00:00 2023-06-23T01:29:16.295 INFO:teuthology.orchestra.run.smithi092.stdout:(63/120): python3-rsa-4.8-1.el8.noarch.rpm 317 kB/s | 61 kB 00:00 2023-06-23T01:29:16.342 INFO:teuthology.orchestra.run.smithi092.stdout:(64/120): python3-trustme-0.6.0-4.el8.noarch.rp 348 kB/s | 27 kB 00:00 2023-06-23T01:29:16.366 INFO:teuthology.orchestra.run.smithi092.stdout:(65/120): python3-waitress-1.4.3-1.el8.noarch.r 2.8 MB/s | 255 kB 00:00 2023-06-23T01:29:16.403 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:29:16.424 INFO:teuthology.orchestra.run.smithi092.stdout:(66/120): python3-websocket-client-0.56.0-5.el8 753 kB/s | 61 kB 00:00 2023-06-23T01:29:16.446 INFO:teuthology.orchestra.run.smithi092.stdout:(67/120): python3-webtest-2.0.33-1.el8.noarch.r 1.0 MB/s | 85 kB 00:00 2023-06-23T01:29:16.502 INFO:teuthology.orchestra.run.smithi092.stdout:(68/120): python3-zc-lockfile-2.0-2.el8.noarch. 296 kB/s | 23 kB 00:00 2023-06-23T01:29:16.507 INFO:teuthology.orchestra.run.smithi092.stdout:(69/120): xmlstarlet-1.6.1-20.el8.x86_64.rpm 15 MB/s | 70 kB 00:00 2023-06-23T01:29:16.536 INFO:teuthology.orchestra.run.smithi092.stdout:(70/120): python3-webob-1.8.5-1.el8.1.noarch.rp 1.0 MB/s | 251 kB 00:00 2023-06-23T01:29:16.572 INFO:teuthology.orchestra.run.smithi092.stdout:(71/120): python3-prettytable-0.7.2-14.el8.noar 1.2 MB/s | 44 kB 00:00 2023-06-23T01:29:16.586 INFO:teuthology.orchestra.run.smithi092.stdout:(72/120): python3-jsonpatch-1.21-2.el8.noarch.r 1.9 MB/s | 27 kB 00:00 2023-06-23T01:29:16.604 INFO:teuthology.orchestra.run.smithi092.stdout:(73/120): thrift-0.13.0-2.el8.x86_64.rpm 11 MB/s | 1.7 MB 00:00 2023-06-23T01:29:16.608 INFO:teuthology.orchestra.run.smithi092.stdout:(74/120): jq-1.5-12.el8.x86_64.rpm 7.3 MB/s | 161 kB 00:00 2023-06-23T01:29:16.631 INFO:teuthology.orchestra.run.smithi092.stdout:(75/120): python3-werkzeug-0.12.2-4.el8.noarch. 20 MB/s | 457 kB 00:00 2023-06-23T01:29:16.645 INFO:teuthology.orchestra.run.smithi092.stdout:(76/120): flex-2.6.1-9.el8.x86_64.rpm 7.8 MB/s | 320 kB 00:00 2023-06-23T01:29:16.659 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:29:16.662 INFO:teuthology.orchestra.run.smithi092.stdout:(77/120): python3-markupsafe-0.23-19.el8.x86_64 2.2 MB/s | 39 kB 00:00 2023-06-23T01:29:16.688 INFO:teuthology.orchestra.run.smithi092.stdout:(78/120): bison-3.0.4-10.el8.x86_64.rpm 26 MB/s | 688 kB 00:00 2023-06-23T01:29:16.706 INFO:teuthology.orchestra.run.smithi092.stdout:(79/120): lttng-ust-2.8.1-11.el8.x86_64.rpm 15 MB/s | 259 kB 00:00 2023-06-23T01:29:16.724 INFO:teuthology.orchestra.run.smithi092.stdout:(80/120): python3-babel-2.5.1-5.el8.noarch.rpm 52 MB/s | 4.8 MB 00:00 2023-06-23T01:29:16.728 INFO:teuthology.orchestra.run.smithi092.stdout:(81/120): python3-numpy-f2py-1.14.3-9.el8.x86_6 10 MB/s | 225 kB 00:00 2023-06-23T01:29:16.743 INFO:teuthology.orchestra.run.smithi092.stdout:(82/120): python3-pyasn1-0.3.7-6.el8.noarch.rpm 6.6 MB/s | 126 kB 00:00 2023-06-23T01:29:16.759 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:29:16.839 INFO:teuthology.orchestra.run.smithi092.stdout:(83/120): python3-scipy-1.0.0-20.module+el8.1.0 44 MB/s | 15 MB 00:00 2023-06-23T01:29:16.843 INFO:teuthology.orchestra.run.smithi092.stdout:(84/120): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-06-23T01:29:16.861 INFO:teuthology.orchestra.run.smithi092.stdout:(85/120): socat-1.7.3.3-2.el8.x86_64.rpm 16 MB/s | 302 kB 00:00 2023-06-23T01:29:16.866 INFO:teuthology.orchestra.run.smithi092.stdout:(86/120): python3-jinja2-2.10.1-2.el8_0.noarch. 20 MB/s | 538 kB 00:00 2023-06-23T01:29:16.887 INFO:teuthology.orchestra.run.smithi092.stdout:(87/120): python3-numpy-1.14.3-9.el8.x86_64.rpm 24 MB/s | 3.7 MB 00:00 2023-06-23T01:29:16.888 INFO:teuthology.orchestra.run.smithi092.stdout:(88/120): python3-mako-1.0.6-13.el8.noarch.rpm 5.9 MB/s | 157 kB 00:00 2023-06-23T01:29:16.889 INFO:teuthology.orchestra.run.smithi092.stdout:(89/120): python3-pyasn1-modules-0.3.7-6.el8.no 4.7 MB/s | 110 kB 00:00 2023-06-23T01:29:16.906 INFO:teuthology.orchestra.run.smithi092.stdout:(90/120): librdkafka-0.11.4-1.el8.x86_64.rpm 19 MB/s | 353 kB 00:00 2023-06-23T01:29:16.907 INFO:teuthology.orchestra.run.smithi092.stdout:(91/120): python3-jsonpointer-1.10-11.el8.noarc 1.1 MB/s | 20 kB 00:00 2023-06-23T01:29:16.913 INFO:teuthology.orchestra.run.smithi092.stdout:(92/120): libtool-2.4.6-25.el8.x86_64.rpm 29 MB/s | 709 kB 00:00 2023-06-23T01:29:16.921 INFO:teuthology.orchestra.run.smithi092.stdout:(93/120): python3-pytz-2017.2-9.el8.noarch.rpm 3.5 MB/s | 54 kB 00:00 2023-06-23T01:29:16.950 INFO:teuthology.orchestra.run.smithi092.stdout:(94/120): automake-1.16.1-7.el8.noarch.rpm 27 MB/s | 713 kB 00:00 2023-06-23T01:29:16.961 INFO:teuthology.orchestra.run.smithi092.stdout:(95/120): libaio-devel-0.3.112-1.el8.x86_64.rpm 1.4 MB/s | 19 kB 00:00 2023-06-23T01:29:16.974 INFO:teuthology.orchestra.run.smithi092.stdout:(96/120): python3-jwt-1.6.1-2.el8.noarch.rpm 3.2 MB/s | 43 kB 00:00 2023-06-23T01:29:16.987 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:29:16.990 INFO:teuthology.orchestra.run.smithi092.stdout:(97/120): openblas-threads-0.3.12-1.el8.x86_64. 56 MB/s | 4.7 MB 00:00 2023-06-23T01:29:16.993 INFO:teuthology.orchestra.run.smithi092.stdout:(98/120): python3-oauthlib-2.1.0-1.el8.noarch.r 8.2 MB/s | 155 kB 00:00 2023-06-23T01:29:17.009 INFO:teuthology.orchestra.run.smithi092.stdout:(99/120): openblas-0.3.12-1.el8.x86_64.rpm 48 MB/s | 4.6 MB 00:00 2023-06-23T01:29:17.010 INFO:teuthology.orchestra.run.smithi092.stdout:(100/120): keyutils-libs-devel-1.5.10-6.el8.x86 2.7 MB/s | 48 kB 00:00 2023-06-23T01:29:17.012 INFO:teuthology.orchestra.run.smithi092.stdout:(101/120): pcre2-utf32-10.32-2.el8.x86_64.rpm 9.9 MB/s | 220 kB 00:00 2023-06-23T01:29:17.026 INFO:teuthology.orchestra.run.smithi092.stdout:(102/120): pcre2-utf16-10.32-2.el8.x86_64.rpm 14 MB/s | 229 kB 00:00 2023-06-23T01:29:17.027 INFO:teuthology.orchestra.run.smithi092.stdout:(103/120): python3-requests-oauthlib-1.0.0-1.el 2.5 MB/s | 43 kB 00:00 2023-06-23T01:29:17.039 INFO:teuthology.orchestra.run.smithi092.stdout:(104/120): libverto-devel-0.3.0-5.el8.x86_64.rp 1.3 MB/s | 18 kB 00:00 2023-06-23T01:29:17.049 INFO:teuthology.orchestra.run.smithi092.stdout:(105/120): pcre2-devel-10.32-2.el8.x86_64.rpm 27 MB/s | 605 kB 00:00 2023-06-23T01:29:17.056 INFO:teuthology.orchestra.run.smithi092.stdout:(106/120): openssl-devel-1.1.1g-15.el8_3.x86_64 54 MB/s | 2.3 MB 00:00 2023-06-23T01:29:17.057 INFO:teuthology.orchestra.run.smithi092.stdout:(107/120): fuse-2.9.7-12.el8.x86_64.rpm 4.5 MB/s | 83 kB 00:00 2023-06-23T01:29:17.064 INFO:teuthology.orchestra.run.smithi092.stdout:(108/120): elfutils-libelf-devel-0.182-3.el8.x8 3.8 MB/s | 59 kB 00:00 2023-06-23T01:29:17.073 INFO:teuthology.orchestra.run.smithi092.stdout:(109/120): libquadmath-8.4.1-1.el8.x86_64.rpm 9.8 MB/s | 169 kB 00:00 2023-06-23T01:29:17.074 INFO:teuthology.orchestra.run.smithi092.stdout:(110/120): libcom_err-devel-1.45.6-1.el8.x86_64 2.3 MB/s | 38 kB 00:00 2023-06-23T01:29:17.079 INFO:teuthology.orchestra.run.smithi092.stdout:(111/120): libsepol-devel-2.9-2.el8.x86_64.rpm 5.7 MB/s | 86 kB 00:00 2023-06-23T01:29:17.091 INFO:teuthology.orchestra.run.smithi092.stdout:(112/120): librabbitmq-0.9.0-3.el8.x86_64.rpm 3.0 MB/s | 47 kB 00:00 2023-06-23T01:29:17.091 INFO:teuthology.orchestra.run.smithi092.stdout:(113/120): zlib-devel-1.2.11-17.el8.x86_64.rpm 3.4 MB/s | 58 kB 00:00 2023-06-23T01:29:17.100 INFO:teuthology.orchestra.run.smithi092.stdout:(114/120): krb5-devel-1.18.2-8.3.el8_4.x86_64.r 27 MB/s | 559 kB 00:00 2023-06-23T01:29:17.106 INFO:teuthology.orchestra.run.smithi092.stdout:(115/120): xfsprogs-devel-5.0.0-8.el8.x86_64.rp 4.7 MB/s | 70 kB 00:00 2023-06-23T01:29:17.111 INFO:teuthology.orchestra.run.smithi092.stdout:(116/120): libgfortran-8.4.1-1.el8.x86_64.rpm 29 MB/s | 639 kB 00:00 2023-06-23T01:29:17.115 INFO:teuthology.orchestra.run.smithi092.stdout:(117/120): libselinux-devel-2.9-5.el8.x86_64.rp 13 MB/s | 200 kB 00:00 2023-06-23T01:29:17.119 INFO:teuthology.orchestra.run.smithi092.stdout:(118/120): libuuid-devel-2.32.1-27.el8.x86_64.r 6.6 MB/s | 97 kB 00:00 2023-06-23T01:29:17.307 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 8/122 2023-06-23T01:29:17.535 INFO:teuthology.orchestra.run.smithi092.stdout:(119/120): librados2-17.2.6-719.g42ddc46e.el8.x 8.2 MB/s | 3.5 MB 00:00 2023-06-23T01:29:17.546 INFO:teuthology.orchestra.run.smithi092.stdout:(120/120): librbd1-17.2.6-719.g42ddc46e.el8.x86 8.3 MB/s | 3.6 MB 00:00 2023-06-23T01:29:17.553 INFO:teuthology.orchestra.run.smithi092.stdout:-------------------------------------------------------------------------------- 2023-06-23T01:29:17.553 INFO:teuthology.orchestra.run.smithi092.stdout:Total 24 MB/s | 201 MB 00:08 2023-06-23T01:29:17.600 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction check 2023-06-23T01:29:17.712 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction check succeeded. 2023-06-23T01:29:17.712 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction test 2023-06-23T01:29:17.975 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T01:29:18.175 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:29:18.287 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:29:18.546 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T01:29:18.708 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T01:29:18.909 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:29:19.046 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:29:19.234 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:29:19.319 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:29:19.431 INFO:teuthology.orchestra.run.smithi092.stdout:Transaction test succeeded. 2023-06-23T01:29:19.436 INFO:teuthology.orchestra.run.smithi092.stdout:Running transaction 2023-06-23T01:29:19.522 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 15/122 2023-06-23T01:29:19.807 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 16/122 2023-06-23T01:29:20.065 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 17/122 2023-06-23T01:29:20.236 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 18/122 2023-06-23T01:29:20.401 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T01:29:20.536 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 19/122 2023-06-23T01:29:20.709 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T01:29:21.285 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 20/122 2023-06-23T01:29:21.567 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T01:29:22.554 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 21/122 2023-06-23T01:29:22.750 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T01:29:23.485 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 22/122 2023-06-23T01:29:23.786 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 23/122 2023-06-23T01:29:24.050 INFO:teuthology.orchestra.run.smithi092.stdout: Preparing : 1/1 2023-06-23T01:29:24.067 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 24/122 2023-06-23T01:29:24.259 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/122 2023-06-23T01:29:24.336 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T01:29:24.515 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T01:29:24.516 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 2/122 2023-06-23T01:29:24.641 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T01:29:24.696 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T01:29:24.832 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 3/122 2023-06-23T01:29:24.916 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:29:25.038 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T01:29:25.180 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 29/122 2023-06-23T01:29:25.424 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/122 2023-06-23T01:29:25.481 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 30/122 2023-06-23T01:29:25.741 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T01:29:26.104 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:29:26.507 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 5/122 2023-06-23T01:29:26.729 INFO:teuthology.orchestra.run.smithi092.stdout: Upgrading : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:29:26.858 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:29:27.071 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:29:27.210 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:29:27.561 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:29:27.813 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 31/122 2023-06-23T01:29:28.180 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 32/122 2023-06-23T01:29:28.231 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : thrift-0.13.0-2.el8.x86_64 9/122 2023-06-23T01:29:28.387 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:29:28.517 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:29:28.776 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/122 2023-06-23T01:29:28.961 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/122 2023-06-23T01:29:29.154 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:29:29.251 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/122 2023-06-23T01:29:29.259 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:29:29.445 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:29:29.513 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 14/122 2023-06-23T01:29:29.653 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 34/122 2023-06-23T01:29:29.669 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : zlib-devel-1.2.11-17.el8.x86_64 15/122 2023-06-23T01:29:29.845 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T01:29:29.928 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 35/122 2023-06-23T01:29:29.963 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libquadmath-8.4.1-1.el8.x86_64 16/122 2023-06-23T01:29:30.128 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T01:29:30.196 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T01:29:30.435 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T01:29:30.680 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libgfortran-8.4.1-1.el8.x86_64 17/122 2023-06-23T01:29:30.694 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T01:29:30.845 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T01:29:30.932 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T01:29:31.234 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T01:29:31.387 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: openblas-threads-0.3.12-1.el8.x86_64 18/122 2023-06-23T01:29:31.489 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : automake-1.16.1-7.el8.noarch 41/122 2023-06-23T01:29:31.706 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 42/122 2023-06-23T01:29:31.811 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T01:29:31.927 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 43/122 2023-06-23T01:29:32.136 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 44/122 2023-06-23T01:29:32.340 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T01:29:32.425 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 45/122 2023-06-23T01:29:32.540 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: openblas-0.3.12-1.el8.x86_64 19/122 2023-06-23T01:29:32.607 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : jq-1.5-12.el8.x86_64 46/122 2023-06-23T01:29:32.806 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 20/122 2023-06-23T01:29:32.809 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 47/122 2023-06-23T01:29:33.015 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 48/122 2023-06-23T01:29:33.052 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 21/122 2023-06-23T01:29:33.186 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 49/122 2023-06-23T01:29:33.232 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 22/122 2023-06-23T01:29:33.383 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 50/122 2023-06-23T01:29:33.526 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/122 2023-06-23T01:29:33.526 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : fuse-2.9.7-12.el8.x86_64 51/122 2023-06-23T01:29:33.694 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 52/122 2023-06-23T01:29:33.805 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 24/122 2023-06-23T01:29:33.916 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 53/122 2023-06-23T01:29:34.031 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/122 2023-06-23T01:29:34.102 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 54/122 2023-06-23T01:29:34.218 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 26/122 2023-06-23T01:29:34.392 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 27/122 2023-06-23T01:29:34.464 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/122 2023-06-23T01:29:34.635 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:29:34.728 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/122 2023-06-23T01:29:35.027 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/122 2023-06-23T01:29:35.351 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 58/122 2023-06-23T01:29:35.557 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T01:29:35.675 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/122 2023-06-23T01:29:35.761 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T01:29:35.983 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 30/122 2023-06-23T01:29:36.055 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T01:29:36.306 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T01:29:36.318 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 31/122 2023-06-23T01:29:36.522 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T01:29:36.593 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 32/122 2023-06-23T01:29:36.719 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T01:29:36.909 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T01:29:37.095 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 33/122 2023-06-23T01:29:37.133 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T01:29:37.318 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T01:29:37.377 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T01:29:37.617 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T01:29:37.834 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T01:29:38.046 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T01:29:38.312 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T01:29:38.552 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T01:29:38.796 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T01:29:39.102 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T01:29:39.397 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T01:29:39.400 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 34/122 2023-06-23T01:29:39.614 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T01:29:39.747 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 35/122 2023-06-23T01:29:39.890 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T01:29:39.965 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rsa-4.8-1.el8.noarch 36/122 2023-06-23T01:29:40.187 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 37/122 2023-06-23T01:29:40.386 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 38/122 2023-06-23T01:29:40.590 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 39/122 2023-06-23T01:29:40.786 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 40/122 2023-06-23T01:29:40.966 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libuuid-devel-2.32.1-27.el8.x86_64 41/122 2023-06-23T01:29:41.038 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T01:29:41.123 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libsepol-devel-2.9-2.el8.x86_64 42/122 2023-06-23T01:29:41.302 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcom_err-devel-1.45.6-1.el8.x86_64 43/122 2023-06-23T01:29:41.404 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T01:29:41.495 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : fuse-2.9.7-12.el8.x86_64 44/122 2023-06-23T01:29:41.649 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T01:29:41.654 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 45/122 2023-06-23T01:29:41.851 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 46/122 2023-06-23T01:29:42.007 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T01:29:42.037 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : keyutils-libs-devel-1.5.10-6.el8.x86_64 47/122 2023-06-23T01:29:42.304 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T01:29:42.332 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 48/122 2023-06-23T01:29:42.596 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 49/122 2023-06-23T01:29:42.654 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T01:29:42.866 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 50/122 2023-06-23T01:29:42.905 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T01:29:43.114 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T01:29:43.177 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : krb5-devel-1.18.2-8.3.el8_4.x86_64 51/122 2023-06-23T01:29:43.332 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T01:29:43.408 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : automake-1.16.1-7.el8.noarch 52/122 2023-06-23T01:29:43.591 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 53/122 2023-06-23T01:29:43.813 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 54/122 2023-06-23T01:29:43.987 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 55/122 2023-06-23T01:29:44.175 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T01:29:44.261 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 56/122 2023-06-23T01:29:44.425 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : jq-1.5-12.el8.x86_64 57/122 2023-06-23T01:29:44.628 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 58/122 2023-06-23T01:29:44.784 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 59/122 2023-06-23T01:29:44.972 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 60/122 2023-06-23T01:29:45.247 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 61/122 2023-06-23T01:29:45.508 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 62/122 2023-06-23T01:29:45.682 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 63/122 2023-06-23T01:29:45.888 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 64/122 2023-06-23T01:29:46.027 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 65/122 2023-06-23T01:29:46.208 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 66/122 2023-06-23T01:29:46.436 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 67/122 2023-06-23T01:29:46.643 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 68/122 2023-06-23T01:29:46.869 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 69/122 2023-06-23T01:29:47.039 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 70/122 2023-06-23T01:29:47.325 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 71/122 2023-06-23T01:29:47.540 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 72/122 2023-06-23T01:29:47.772 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 73/122 2023-06-23T01:29:48.061 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-portend-2.6-1.el8.noarch 74/122 2023-06-23T01:29:48.232 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:29:48.324 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 75/122 2023-06-23T01:29:48.532 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 76/122 2023-06-23T01:29:48.797 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 77/122 2023-06-23T01:29:49.346 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:29:49.946 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:29:49.951 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 78/122 2023-06-23T01:29:50.238 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:29:50.238 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 79/122 2023-06-23T01:29:50.339 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:29:50.484 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 80/122 2023-06-23T01:29:50.504 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:29:50.798 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 81/122 2023-06-23T01:29:51.057 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 82/122 2023-06-23T01:29:51.412 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 83/122 2023-06-23T01:29:51.646 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 84/122 2023-06-23T01:29:51.866 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 85/122 2023-06-23T01:29:52.066 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 86/122 2023-06-23T01:29:57.216 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:29:57.465 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:29:58.067 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 87/122 2023-06-23T01:29:59.053 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:29:59.227 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 88/122 2023-06-23T01:29:59.372 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:30:08.882 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:30:08.882 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /sys 2023-06-23T01:30:08.882 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /proc 2023-06-23T01:30:08.883 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /dev 2023-06-23T01:30:08.883 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /run 2023-06-23T01:30:08.883 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /mnt 2023-06-23T01:30:08.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /var/tmp 2023-06-23T01:30:08.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /home 2023-06-23T01:30:08.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /tmp 2023-06-23T01:30:08.884 INFO:teuthology.orchestra.run.smithi119.stdout:skipping the directory /dev 2023-06-23T01:30:08.885 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:09.301 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:30:10.792 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:30:10.934 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:30:11.169 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:30:11.343 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:30:11.732 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:30:11.917 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:30:11.934 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:30:14.643 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:30:14.852 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:30:15.015 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T01:30:15.231 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T01:30:15.449 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:30:15.666 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:30:15.908 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:30:16.876 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:30:17.066 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:30:17.165 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:30:17.244 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 89/122 2023-06-23T01:30:17.245 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /sys 2023-06-23T01:30:17.245 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /proc 2023-06-23T01:30:17.245 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /dev 2023-06-23T01:30:17.245 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /run 2023-06-23T01:30:17.246 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /mnt 2023-06-23T01:30:17.246 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /var/tmp 2023-06-23T01:30:17.246 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /home 2023-06-23T01:30:17.246 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /tmp 2023-06-23T01:30:17.247 INFO:teuthology.orchestra.run.smithi092.stdout:skipping the directory /dev 2023-06-23T01:30:17.247 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:17.539 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T01:30:17.651 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:30:17.741 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:30:17.780 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:30:17.966 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:30:17.977 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:30:19.134 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 90/122 2023-06-23T01:30:19.306 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:30:19.536 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 91/122 2023-06-23T01:30:19.595 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T01:30:19.710 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:30:19.849 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:30:19.937 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:30:20.098 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 92/122 2023-06-23T01:30:20.107 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:30:20.156 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:30:20.259 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:30:20.275 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 93/122 2023-06-23T01:30:20.324 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:30:20.675 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:30:20.824 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:30:22.947 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:30:23.081 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 94/122 2023-06-23T01:30:23.298 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 95/122 2023-06-23T01:30:23.488 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 96/122 2023-06-23T01:30:23.683 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:30:23.897 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 97/122 2023-06-23T01:30:24.129 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:30:25.101 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 98/122 2023-06-23T01:30:25.291 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:30:25.391 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 99/122 2023-06-23T01:30:25.751 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 100/122 2023-06-23T01:30:25.809 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:30:25.938 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:30:25.974 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 101/122 2023-06-23T01:30:26.141 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:30:26.152 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 102/122 2023-06-23T01:30:26.778 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T01:30:27.215 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 109/122 2023-06-23T01:30:27.431 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 110/122 2023-06-23T01:30:27.684 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 111/122 2023-06-23T01:30:27.780 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 103/122 2023-06-23T01:30:27.881 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T01:30:27.892 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 112/122 2023-06-23T01:30:27.979 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:30:28.086 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T01:30:28.086 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 104/122 2023-06-23T01:30:28.228 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T01:30:28.232 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:30:28.275 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 105/122 2023-06-23T01:30:28.406 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T01:30:28.415 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:30:28.587 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T01:30:28.764 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 106/122 2023-06-23T01:30:28.849 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T01:30:28.957 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:30:29.056 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T01:30:29.117 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 118/122 2023-06-23T01:30:29.322 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T01:30:29.331 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 119/122 2023-06-23T01:30:29.483 INFO:teuthology.orchestra.run.smithi119.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 120/122 2023-06-23T01:30:29.483 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:30:29.708 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:30:29.709 INFO:teuthology.orchestra.run.smithi119.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:30:30.253 INFO:teuthology.orchestra.run.smithi119.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:30:30.253 INFO:teuthology.orchestra.run.smithi119.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T01:30:30.254 INFO:teuthology.orchestra.run.smithi119.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T01:30:30.254 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:30.603 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T01:30:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T01:30:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T01:30:30.604 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:30:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T01:30:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:30:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:30:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:30:30.605 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T01:30:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:30:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T01:30:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T01:30:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:30:30.606 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T01:30:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T01:30:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T01:30:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T01:30:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T01:30:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T01:30:30.607 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T01:30:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T01:30:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T01:30:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T01:30:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T01:30:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T01:30:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T01:30:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T01:30:30.608 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:30:30.609 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T01:30:30.609 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T01:30:30.609 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T01:30:30.609 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T01:30:30.609 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T01:30:30.609 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T01:30:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T01:30:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T01:30:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T01:30:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T01:30:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T01:30:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T01:30:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T01:30:30.610 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T01:30:30.611 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T01:30:30.611 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T01:30:30.611 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T01:30:30.611 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T01:30:30.611 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T01:30:30.611 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T01:30:30.611 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T01:30:30.611 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T01:30:30.612 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T01:30:30.612 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T01:30:30.613 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T01:30:30.613 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T01:30:30.613 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T01:30:30.614 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T01:30:30.614 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T01:30:30.614 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T01:30:30.615 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T01:30:30.615 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T01:30:30.615 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T01:30:30.616 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T01:30:30.616 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T01:30:30.616 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T01:30:30.616 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T01:30:30.616 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T01:30:30.617 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T01:30:30.617 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T01:30:30.617 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T01:30:30.617 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T01:30:30.617 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T01:30:30.618 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 72/122 2023-06-23T01:30:30.618 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 73/122 2023-06-23T01:30:30.618 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 74/122 2023-06-23T01:30:30.618 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/122 2023-06-23T01:30:30.619 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 76/122 2023-06-23T01:30:30.619 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 77/122 2023-06-23T01:30:30.619 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/122 2023-06-23T01:30:30.619 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 79/122 2023-06-23T01:30:30.619 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 80/122 2023-06-23T01:30:30.619 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 81/122 2023-06-23T01:30:30.619 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 82/122 2023-06-23T01:30:30.619 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 83/122 2023-06-23T01:30:30.620 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 84/122 2023-06-23T01:30:30.620 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 85/122 2023-06-23T01:30:30.620 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 86/122 2023-06-23T01:30:30.620 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 87/122 2023-06-23T01:30:30.620 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 88/122 2023-06-23T01:30:30.620 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 89/122 2023-06-23T01:30:30.620 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 90/122 2023-06-23T01:30:30.620 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 91/122 2023-06-23T01:30:30.621 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 92/122 2023-06-23T01:30:30.621 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 93/122 2023-06-23T01:30:30.621 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 94/122 2023-06-23T01:30:30.621 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 95/122 2023-06-23T01:30:30.621 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 96/122 2023-06-23T01:30:30.621 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : jq-1.5-12.el8.x86_64 97/122 2023-06-23T01:30:30.621 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : flex-2.6.1-9.el8.x86_64 98/122 2023-06-23T01:30:30.622 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 99/122 2023-06-23T01:30:30.622 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 100/122 2023-06-23T01:30:30.622 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 101/122 2023-06-23T01:30:30.622 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : bison-3.0.4-10.el8.x86_64 102/122 2023-06-23T01:30:30.622 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 103/122 2023-06-23T01:30:30.622 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 104/122 2023-06-23T01:30:30.622 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 105/122 2023-06-23T01:30:30.622 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 106/122 2023-06-23T01:30:30.623 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/122 2023-06-23T01:30:30.623 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 108/122 2023-06-23T01:30:30.623 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 109/122 2023-06-23T01:30:30.623 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 110/122 2023-06-23T01:30:30.623 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 111/122 2023-06-23T01:30:30.624 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 112/122 2023-06-23T01:30:30.624 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 113/122 2023-06-23T01:30:30.624 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 114/122 2023-06-23T01:30:30.624 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 115/122 2023-06-23T01:30:30.624 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 116/122 2023-06-23T01:30:30.624 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 117/122 2023-06-23T01:30:30.624 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : automake-1.16.1-7.el8.noarch 118/122 2023-06-23T01:30:30.625 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T01:30:30.625 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T01:30:30.625 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T01:30:32.908 INFO:teuthology.orchestra.run.smithi119.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:30:32.909 INFO:teuthology.orchestra.run.smithi119.stdout:Last metadata expiration check: 0:01:44 ago on Fri 23 Jun 2023 01:28:48 AM UTC. 2023-06-23T01:30:33.912 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 107/122 2023-06-23T01:30:34.222 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 108/122 2023-06-23T01:30:34.389 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T01:30:35.092 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 109/122 2023-06-23T01:30:35.248 INFO:teuthology.orchestra.run.smithi119.stdout:Installed products updated. 2023-06-23T01:30:35.424 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:35.424 INFO:teuthology.orchestra.run.smithi119.stdout:Upgraded: 2023-06-23T01:30:35.424 INFO:teuthology.orchestra.run.smithi119.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.425 INFO:teuthology.orchestra.run.smithi119.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.425 INFO:teuthology.orchestra.run.smithi119.stdout:Installed: 2023-06-23T01:30:35.425 INFO:teuthology.orchestra.run.smithi119.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T01:30:35.425 INFO:teuthology.orchestra.run.smithi119.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T01:30:35.425 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.425 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.425 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.426 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.426 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.426 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.426 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.426 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.426 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.426 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.426 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.427 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.427 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.427 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.427 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.427 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.427 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.427 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.428 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.428 INFO:teuthology.orchestra.run.smithi119.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.428 INFO:teuthology.orchestra.run.smithi119.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.428 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.428 INFO:teuthology.orchestra.run.smithi119.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:35.428 INFO:teuthology.orchestra.run.smithi119.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T01:30:35.428 INFO:teuthology.orchestra.run.smithi119.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T01:30:35.429 INFO:teuthology.orchestra.run.smithi119.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T01:30:35.429 INFO:teuthology.orchestra.run.smithi119.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T01:30:35.429 INFO:teuthology.orchestra.run.smithi119.stdout: jq-1.5-12.el8.x86_64 2023-06-23T01:30:35.429 INFO:teuthology.orchestra.run.smithi119.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T01:30:35.429 INFO:teuthology.orchestra.run.smithi119.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T01:30:35.429 INFO:teuthology.orchestra.run.smithi119.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T01:30:35.430 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.430 INFO:teuthology.orchestra.run.smithi119.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.430 INFO:teuthology.orchestra.run.smithi119.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.430 INFO:teuthology.orchestra.run.smithi119.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T01:30:35.430 INFO:teuthology.orchestra.run.smithi119.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T01:30:35.430 INFO:teuthology.orchestra.run.smithi119.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T01:30:35.430 INFO:teuthology.orchestra.run.smithi119.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T01:30:35.431 INFO:teuthology.orchestra.run.smithi119.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T01:30:35.431 INFO:teuthology.orchestra.run.smithi119.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.431 INFO:teuthology.orchestra.run.smithi119.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.431 INFO:teuthology.orchestra.run.smithi119.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T01:30:35.431 INFO:teuthology.orchestra.run.smithi119.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.431 INFO:teuthology.orchestra.run.smithi119.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T01:30:35.432 INFO:teuthology.orchestra.run.smithi119.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T01:30:35.432 INFO:teuthology.orchestra.run.smithi119.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T01:30:35.432 INFO:teuthology.orchestra.run.smithi119.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T01:30:35.432 INFO:teuthology.orchestra.run.smithi119.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T01:30:35.432 INFO:teuthology.orchestra.run.smithi119.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T01:30:35.433 INFO:teuthology.orchestra.run.smithi119.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T01:30:35.433 INFO:teuthology.orchestra.run.smithi119.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T01:30:35.433 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T01:30:35.433 INFO:teuthology.orchestra.run.smithi119.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T01:30:35.433 INFO:teuthology.orchestra.run.smithi119.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T01:30:35.434 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T01:30:35.434 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T01:30:35.434 INFO:teuthology.orchestra.run.smithi119.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T01:30:35.434 INFO:teuthology.orchestra.run.smithi119.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T01:30:35.434 INFO:teuthology.orchestra.run.smithi119.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T01:30:35.434 INFO:teuthology.orchestra.run.smithi119.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T01:30:35.434 INFO:teuthology.orchestra.run.smithi119.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T01:30:35.435 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T01:30:35.435 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.435 INFO:teuthology.orchestra.run.smithi119.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.435 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.435 INFO:teuthology.orchestra.run.smithi119.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T01:30:35.435 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T01:30:35.435 INFO:teuthology.orchestra.run.smithi119.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T01:30:35.436 INFO:teuthology.orchestra.run.smithi119.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T01:30:35.436 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T01:30:35.436 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T01:30:35.436 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T01:30:35.436 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T01:30:35.436 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T01:30:35.436 INFO:teuthology.orchestra.run.smithi119.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T01:30:35.436 INFO:teuthology.orchestra.run.smithi119.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T01:30:35.437 INFO:teuthology.orchestra.run.smithi119.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T01:30:35.437 INFO:teuthology.orchestra.run.smithi119.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T01:30:35.437 INFO:teuthology.orchestra.run.smithi119.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T01:30:35.437 INFO:teuthology.orchestra.run.smithi119.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T01:30:35.437 INFO:teuthology.orchestra.run.smithi119.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T01:30:35.437 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T01:30:35.437 INFO:teuthology.orchestra.run.smithi119.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T01:30:35.438 INFO:teuthology.orchestra.run.smithi119.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T01:30:35.438 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T01:30:35.438 INFO:teuthology.orchestra.run.smithi119.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T01:30:35.438 INFO:teuthology.orchestra.run.smithi119.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T01:30:35.438 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T01:30:35.438 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T01:30:35.438 INFO:teuthology.orchestra.run.smithi119.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T01:30:35.438 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.439 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.439 INFO:teuthology.orchestra.run.smithi119.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T01:30:35.439 INFO:teuthology.orchestra.run.smithi119.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T01:30:35.439 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.439 INFO:teuthology.orchestra.run.smithi119.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T01:30:35.439 INFO:teuthology.orchestra.run.smithi119.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T01:30:35.439 INFO:teuthology.orchestra.run.smithi119.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T01:30:35.440 INFO:teuthology.orchestra.run.smithi119.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T01:30:35.440 INFO:teuthology.orchestra.run.smithi119.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T01:30:35.440 INFO:teuthology.orchestra.run.smithi119.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T01:30:35.440 INFO:teuthology.orchestra.run.smithi119.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T01:30:35.440 INFO:teuthology.orchestra.run.smithi119.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T01:30:35.441 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T01:30:35.441 INFO:teuthology.orchestra.run.smithi119.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T01:30:35.441 INFO:teuthology.orchestra.run.smithi119.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T01:30:35.441 INFO:teuthology.orchestra.run.smithi119.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T01:30:35.441 INFO:teuthology.orchestra.run.smithi119.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T01:30:35.441 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.441 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.442 INFO:teuthology.orchestra.run.smithi119.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:35.442 INFO:teuthology.orchestra.run.smithi119.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T01:30:35.442 INFO:teuthology.orchestra.run.smithi119.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T01:30:35.442 INFO:teuthology.orchestra.run.smithi119.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T01:30:35.442 INFO:teuthology.orchestra.run.smithi119.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T01:30:35.443 INFO:teuthology.orchestra.run.smithi119.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T01:30:35.443 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:35.443 INFO:teuthology.orchestra.run.smithi119.stdout:Complete! 2023-06-23T01:30:35.447 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : openssl-devel-1:1.1.1g-15.el8_3.x86_64 110/122 2023-06-23T01:30:35.647 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 111/122 2023-06-23T01:30:35.716 DEBUG:teuthology.parallel:result is None 2023-06-23T01:30:35.792 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : xfsprogs-devel-5.0.0-8.el8.x86_64 112/122 2023-06-23T01:30:35.935 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 113/122 2023-06-23T01:30:36.110 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : elfutils-libelf-devel-0.182-3.el8.x86_64 114/122 2023-06-23T01:30:36.264 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 115/122 2023-06-23T01:30:36.492 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 116/122 2023-06-23T01:30:36.640 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 117/122 2023-06-23T01:30:36.863 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 118/122 2023-06-23T01:30:37.088 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T01:30:37.146 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 119/122 2023-06-23T01:30:37.355 INFO:teuthology.orchestra.run.smithi092.stdout: Installing : flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T01:30:37.364 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 120/122 2023-06-23T01:30:37.364 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:30:37.575 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 121/122 2023-06-23T01:30:37.575 INFO:teuthology.orchestra.run.smithi092.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:30:38.149 INFO:teuthology.orchestra.run.smithi092.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:30:38.149 INFO:teuthology.orchestra.run.smithi092.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-06-23T01:30:38.150 INFO:teuthology.orchestra.run.smithi092.stdout:All rules containing unresolvable specifiers will be skipped. 2023-06-23T01:30:38.150 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 1/122 2023-06-23T01:30:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2/122 2023-06-23T01:30:38.471 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 3/122 2023-06-23T01:30:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 4/122 2023-06-23T01:30:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-719.g42ddc4 5/122 2023-06-23T01:30:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 6/122 2023-06-23T01:30:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 7/122 2023-06-23T01:30:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 8/122 2023-06-23T01:30:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 9/122 2023-06-23T01:30:38.472 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 10/122 2023-06-23T01:30:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 11/122 2023-06-23T01:30:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 12/122 2023-06-23T01:30:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 13/122 2023-06-23T01:30:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_6 14/122 2023-06-23T01:30:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 15/122 2023-06-23T01:30:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 16/122 2023-06-23T01:30:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 17/122 2023-06-23T01:30:38.473 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_ 18/122 2023-06-23T01:30:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 19/122 2023-06-23T01:30:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8 20/122 2023-06-23T01:30:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x 21/122 2023-06-23T01:30:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 22/122 2023-06-23T01:30:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 23/122 2023-06-23T01:30:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 24/122 2023-06-23T01:30:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 25/122 2023-06-23T01:30:38.474 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 26/122 2023-06-23T01:30:38.475 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 27/122 2023-06-23T01:30:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 28/122 2023-06-23T01:30:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.e 29/122 2023-06-23T01:30:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noar 30/122 2023-06-23T01:30:38.481 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.no 31/122 2023-06-23T01:30:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-719.g42dd 32/122 2023-06-23T01:30:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8 33/122 2023-06-23T01:30:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 34/122 2023-06-23T01:30:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el 35/122 2023-06-23T01:30:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 36/122 2023-06-23T01:30:38.482 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 37/122 2023-06-23T01:30:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 38/122 2023-06-23T01:30:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/122 2023-06-23T01:30:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/122 2023-06-23T01:30:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/122 2023-06-23T01:30:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/122 2023-06-23T01:30:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/122 2023-06-23T01:30:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/122 2023-06-23T01:30:38.483 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/122 2023-06-23T01:30:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/122 2023-06-23T01:30:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/122 2023-06-23T01:30:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/122 2023-06-23T01:30:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/122 2023-06-23T01:30:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/122 2023-06-23T01:30:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/122 2023-06-23T01:30:38.484 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/122 2023-06-23T01:30:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/122 2023-06-23T01:30:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/122 2023-06-23T01:30:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/122 2023-06-23T01:30:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/122 2023-06-23T01:30:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/122 2023-06-23T01:30:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/122 2023-06-23T01:30:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/122 2023-06-23T01:30:38.485 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/122 2023-06-23T01:30:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/122 2023-06-23T01:30:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/122 2023-06-23T01:30:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/122 2023-06-23T01:30:38.486 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/122 2023-06-23T01:30:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 65/122 2023-06-23T01:30:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/122 2023-06-23T01:30:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/122 2023-06-23T01:30:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/122 2023-06-23T01:30:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/122 2023-06-23T01:30:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/122 2023-06-23T01:30:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/122 2023-06-23T01:30:38.487 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb62 72/122 2023-06-23T01:30:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 73/122 2023-06-23T01:30:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 74/122 2023-06-23T01:30:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : jq-1.5-12.el8.x86_64 75/122 2023-06-23T01:30:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : flex-2.6.1-9.el8.x86_64 76/122 2023-06-23T01:30:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 77/122 2023-06-23T01:30:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 78/122 2023-06-23T01:30:38.488 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 79/122 2023-06-23T01:30:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : bison-3.0.4-10.el8.x86_64 80/122 2023-06-23T01:30:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 81/122 2023-06-23T01:30:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 82/122 2023-06-23T01:30:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 83/122 2023-06-23T01:30:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 84/122 2023-06-23T01:30:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 85/122 2023-06-23T01:30:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 86/122 2023-06-23T01:30:38.489 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 87/122 2023-06-23T01:30:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 88/122 2023-06-23T01:30:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 89/122 2023-06-23T01:30:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 90/122 2023-06-23T01:30:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 91/122 2023-06-23T01:30:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 92/122 2023-06-23T01:30:38.490 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 93/122 2023-06-23T01:30:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-threads-0.3.12-1.el8.x86_64 94/122 2023-06-23T01:30:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openblas-0.3.12-1.el8.x86_64 95/122 2023-06-23T01:30:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : automake-1.16.1-7.el8.noarch 96/122 2023-06-23T01:30:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 97/122 2023-06-23T01:30:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 98/122 2023-06-23T01:30:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 99/122 2023-06-23T01:30:38.491 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 100/122 2023-06-23T01:30:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : keyutils-libs-devel-1.5.10-6.el8.x86_64 101/122 2023-06-23T01:30:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 102/122 2023-06-23T01:30:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 103/122 2023-06-23T01:30:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : openssl-devel-1:1.1.1g-15.el8_3.x86_64 104/122 2023-06-23T01:30:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 105/122 2023-06-23T01:30:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 106/122 2023-06-23T01:30:38.492 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 107/122 2023-06-23T01:30:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : elfutils-libelf-devel-0.182-3.el8.x86_64 108/122 2023-06-23T01:30:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libquadmath-8.4.1-1.el8.x86_64 109/122 2023-06-23T01:30:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libcom_err-devel-1.45.6-1.el8.x86_64 110/122 2023-06-23T01:30:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libsepol-devel-2.9-2.el8.x86_64 111/122 2023-06-23T01:30:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : zlib-devel-1.2.11-17.el8.x86_64 112/122 2023-06-23T01:30:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 113/122 2023-06-23T01:30:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : krb5-devel-1.18.2-8.3.el8_4.x86_64 114/122 2023-06-23T01:30:38.493 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libgfortran-8.4.1-1.el8.x86_64 115/122 2023-06-23T01:30:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : xfsprogs-devel-5.0.0-8.el8.x86_64 116/122 2023-06-23T01:30:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 117/122 2023-06-23T01:30:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : libuuid-devel-2.32.1-27.el8.x86_64 118/122 2023-06-23T01:30:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 119/122 2023-06-23T01:30:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 120/122 2023-06-23T01:30:38.494 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 121/122 2023-06-23T01:30:40.746 INFO:teuthology.orchestra.run.smithi092.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 122/122 2023-06-23T01:30:40.747 INFO:teuthology.orchestra.run.smithi092.stdout:Last metadata expiration check: 0:01:40 ago on Fri 23 Jun 2023 01:29:00 AM UTC. 2023-06-23T01:30:42.942 INFO:teuthology.orchestra.run.smithi092.stdout:Installed products updated. 2023-06-23T01:30:43.123 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:43.124 INFO:teuthology.orchestra.run.smithi092.stdout:Upgraded: 2023-06-23T01:30:43.124 INFO:teuthology.orchestra.run.smithi092.stdout: librados2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.124 INFO:teuthology.orchestra.run.smithi092.stdout: librbd1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.124 INFO:teuthology.orchestra.run.smithi092.stdout:Installed: 2023-06-23T01:30:43.124 INFO:teuthology.orchestra.run.smithi092.stdout: automake-1.16.1-7.el8.noarch 2023-06-23T01:30:43.124 INFO:teuthology.orchestra.run.smithi092.stdout: bison-3.0.4-10.el8.x86_64 2023-06-23T01:30:43.124 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.125 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-base-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.125 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.125 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.125 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-grafana-dashboards-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.125 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-immutable-object-cache-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.125 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mds-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.125 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.126 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.126 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-dashboard-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.126 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-diskprediction-local-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.126 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-modules-core-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.126 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mgr-rook-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.126 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-mon-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.126 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-osd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.127 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-prometheus-alerts-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.127 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-radosgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.127 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-selinux-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.127 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-test-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.128 INFO:teuthology.orchestra.run.smithi092.stdout: ceph-volume-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.128 INFO:teuthology.orchestra.run.smithi092.stdout: cephadm-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.128 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.129 INFO:teuthology.orchestra.run.smithi092.stdout: cephfs-top-2:17.2.6-719.g42ddc46e.el8.noarch 2023-06-23T01:30:43.129 INFO:teuthology.orchestra.run.smithi092.stdout: elfutils-libelf-devel-0.182-3.el8.x86_64 2023-06-23T01:30:43.129 INFO:teuthology.orchestra.run.smithi092.stdout: flex-2.6.1-9.el8.x86_64 2023-06-23T01:30:43.129 INFO:teuthology.orchestra.run.smithi092.stdout: fuse-2.9.7-12.el8.x86_64 2023-06-23T01:30:43.130 INFO:teuthology.orchestra.run.smithi092.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-06-23T01:30:43.130 INFO:teuthology.orchestra.run.smithi092.stdout: jq-1.5-12.el8.x86_64 2023-06-23T01:30:43.130 INFO:teuthology.orchestra.run.smithi092.stdout: keyutils-libs-devel-1.5.10-6.el8.x86_64 2023-06-23T01:30:43.130 INFO:teuthology.orchestra.run.smithi092.stdout: krb5-devel-1.18.2-8.3.el8_4.x86_64 2023-06-23T01:30:43.131 INFO:teuthology.orchestra.run.smithi092.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-06-23T01:30:43.131 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.131 INFO:teuthology.orchestra.run.smithi092.stdout: libcephfs2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.132 INFO:teuthology.orchestra.run.smithi092.stdout: libcephsqlite-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.132 INFO:teuthology.orchestra.run.smithi092.stdout: libcom_err-devel-1.45.6-1.el8.x86_64 2023-06-23T01:30:43.132 INFO:teuthology.orchestra.run.smithi092.stdout: libgfortran-8.4.1-1.el8.x86_64 2023-06-23T01:30:43.132 INFO:teuthology.orchestra.run.smithi092.stdout: liboath-2.6.2-3.el8.x86_64 2023-06-23T01:30:43.132 INFO:teuthology.orchestra.run.smithi092.stdout: libquadmath-8.4.1-1.el8.x86_64 2023-06-23T01:30:43.133 INFO:teuthology.orchestra.run.smithi092.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-06-23T01:30:43.133 INFO:teuthology.orchestra.run.smithi092.stdout: librados-devel-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.133 INFO:teuthology.orchestra.run.smithi092.stdout: libradosstriper1-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.133 INFO:teuthology.orchestra.run.smithi092.stdout: librdkafka-0.11.4-1.el8.x86_64 2023-06-23T01:30:43.134 INFO:teuthology.orchestra.run.smithi092.stdout: librgw2-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.134 INFO:teuthology.orchestra.run.smithi092.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-06-23T01:30:43.134 INFO:teuthology.orchestra.run.smithi092.stdout: libsepol-devel-2.9-2.el8.x86_64 2023-06-23T01:30:43.134 INFO:teuthology.orchestra.run.smithi092.stdout: libtool-2.4.6-25.el8.x86_64 2023-06-23T01:30:43.134 INFO:teuthology.orchestra.run.smithi092.stdout: libunwind-1.3.1-3.el8.x86_64 2023-06-23T01:30:43.134 INFO:teuthology.orchestra.run.smithi092.stdout: libuuid-devel-2.32.1-27.el8.x86_64 2023-06-23T01:30:43.134 INFO:teuthology.orchestra.run.smithi092.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-06-23T01:30:43.135 INFO:teuthology.orchestra.run.smithi092.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-06-23T01:30:43.135 INFO:teuthology.orchestra.run.smithi092.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-06-23T01:30:43.135 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-0.3.12-1.el8.x86_64 2023-06-23T01:30:43.135 INFO:teuthology.orchestra.run.smithi092.stdout: openblas-threads-0.3.12-1.el8.x86_64 2023-06-23T01:30:43.135 INFO:teuthology.orchestra.run.smithi092.stdout: openssl-devel-1:1.1.1g-15.el8_3.x86_64 2023-06-23T01:30:43.135 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-06-23T01:30:43.135 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-06-23T01:30:43.135 INFO:teuthology.orchestra.run.smithi092.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-06-23T01:30:43.136 INFO:teuthology.orchestra.run.smithi092.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-06-23T01:30:43.136 INFO:teuthology.orchestra.run.smithi092.stdout: python3-babel-2.5.1-5.el8.noarch 2023-06-23T01:30:43.136 INFO:teuthology.orchestra.run.smithi092.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-06-23T01:30:43.136 INFO:teuthology.orchestra.run.smithi092.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-06-23T01:30:43.136 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-06-23T01:30:43.136 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-argparse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.136 INFO:teuthology.orchestra.run.smithi092.stdout: python3-ceph-common-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.137 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cephfs-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.137 INFO:teuthology.orchestra.run.smithi092.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-06-23T01:30:43.137 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-06-23T01:30:43.137 INFO:teuthology.orchestra.run.smithi092.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-06-23T01:30:43.137 INFO:teuthology.orchestra.run.smithi092.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-06-23T01:30:43.137 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-6.2-6.el8.noarch 2023-06-23T01:30:43.137 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-06-23T01:30:43.137 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2023-06-23T01:30:43.138 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-06-23T01:30:43.138 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-06-23T01:30:43.138 INFO:teuthology.orchestra.run.smithi092.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-06-23T01:30:43.138 INFO:teuthology.orchestra.run.smithi092.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-06-23T01:30:43.138 INFO:teuthology.orchestra.run.smithi092.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-06-23T01:30:43.138 INFO:teuthology.orchestra.run.smithi092.stdout: python3-mako-1.0.6-13.el8.noarch 2023-06-23T01:30:43.138 INFO:teuthology.orchestra.run.smithi092.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-06-23T01:30:43.138 INFO:teuthology.orchestra.run.smithi092.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-06-23T01:30:43.139 INFO:teuthology.orchestra.run.smithi092.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-06-23T01:30:43.139 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2023-06-23T01:30:43.139 INFO:teuthology.orchestra.run.smithi092.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2023-06-23T01:30:43.139 INFO:teuthology.orchestra.run.smithi092.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-06-23T01:30:43.139 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-06-23T01:30:43.139 INFO:teuthology.orchestra.run.smithi092.stdout: python3-portend-2.6-1.el8.noarch 2023-06-23T01:30:43.139 INFO:teuthology.orchestra.run.smithi092.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-06-23T01:30:43.140 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-06-23T01:30:43.140 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-06-23T01:30:43.140 INFO:teuthology.orchestra.run.smithi092.stdout: python3-pytz-2017.2-9.el8.noarch 2023-06-23T01:30:43.140 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rados-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.140 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.140 INFO:teuthology.orchestra.run.smithi092.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-06-23T01:30:43.140 INFO:teuthology.orchestra.run.smithi092.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-06-23T01:30:43.140 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rgw-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.141 INFO:teuthology.orchestra.run.smithi092.stdout: python3-routes-2.4.1-12.el8.noarch 2023-06-23T01:30:43.141 INFO:teuthology.orchestra.run.smithi092.stdout: python3-rsa-4.8-1.el8.noarch 2023-06-23T01:30:43.141 INFO:teuthology.orchestra.run.smithi092.stdout: python3-scipy-1.0.0-20.module+el8.1.0+3334+5cb623d7.x86_64 2023-06-23T01:30:43.141 INFO:teuthology.orchestra.run.smithi092.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-06-23T01:30:43.141 INFO:teuthology.orchestra.run.smithi092.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-06-23T01:30:43.141 INFO:teuthology.orchestra.run.smithi092.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-06-23T01:30:43.141 INFO:teuthology.orchestra.run.smithi092.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-06-23T01:30:43.142 INFO:teuthology.orchestra.run.smithi092.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-06-23T01:30:43.142 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-06-23T01:30:43.142 INFO:teuthology.orchestra.run.smithi092.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-06-23T01:30:43.142 INFO:teuthology.orchestra.run.smithi092.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-06-23T01:30:43.142 INFO:teuthology.orchestra.run.smithi092.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-06-23T01:30:43.142 INFO:teuthology.orchestra.run.smithi092.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-06-23T01:30:43.142 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-fuse-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.142 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-mirror-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.143 INFO:teuthology.orchestra.run.smithi092.stdout: rbd-nbd-2:17.2.6-719.g42ddc46e.el8.x86_64 2023-06-23T01:30:43.143 INFO:teuthology.orchestra.run.smithi092.stdout: socat-1.7.3.3-2.el8.x86_64 2023-06-23T01:30:43.143 INFO:teuthology.orchestra.run.smithi092.stdout: thrift-0.13.0-2.el8.x86_64 2023-06-23T01:30:43.143 INFO:teuthology.orchestra.run.smithi092.stdout: xfsprogs-devel-5.0.0-8.el8.x86_64 2023-06-23T01:30:43.143 INFO:teuthology.orchestra.run.smithi092.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-06-23T01:30:43.144 INFO:teuthology.orchestra.run.smithi092.stdout: zlib-devel-1.2.11-17.el8.x86_64 2023-06-23T01:30:43.144 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:43.144 INFO:teuthology.orchestra.run.smithi092.stdout:Complete! 2023-06-23T01:30:43.415 DEBUG:teuthology.parallel:result is None 2023-06-23T01:30:43.416 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:30:43.642 DEBUG:teuthology.orchestra.run.smithi092:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T01:30:43.664 INFO:teuthology.orchestra.run.smithi092.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T01:30:43.664 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T01:30:43.665 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T01:30:43.665 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=42ddc46e6e6582d8e29a5a982a81886fe028be2c 2023-06-23T01:30:43.799 DEBUG:teuthology.orchestra.run.smithi119:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-06-23T01:30:43.823 INFO:teuthology.orchestra.run.smithi119.stdout:17.2.6-719.g42ddc46e.el8 2023-06-23T01:30:43.824 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-719.g42ddc46e.el8 2023-06-23T01:30:43.824 INFO:teuthology.task.install:The correct ceph version 17.2.6-719.g42ddc46e is installed. 2023-06-23T01:30:43.825 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-06-23T01:30:43.825 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:43.826 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T01:30:43.863 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:30:43.863 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-06-23T01:30:43.905 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-06-23T01:30:43.906 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:43.906 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T01:30:43.930 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T01:30:43.994 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:30:43.995 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/daemon-helper 2023-06-23T01:30:44.022 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-06-23T01:30:44.088 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-06-23T01:30:44.088 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:44.088 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T01:30:44.112 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T01:30:44.175 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:30:44.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/adjust-ulimits 2023-06-23T01:30:44.201 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-06-23T01:30:44.265 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-06-23T01:30:44.266 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:44.266 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T01:30:44.289 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T01:30:44.353 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:30:44.353 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/usr/bin/stdin-killer 2023-06-23T01:30:44.377 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-06-23T01:30:44.495 INFO:teuthology.run_tasks:Running task ceph... 2023-06-23T01:30:44.572 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-06-23T01:30:44.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 777 /var/log/ceph 2023-06-23T01:30:44.575 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 777 /var/log/ceph 2023-06-23T01:30:44.599 INFO:tasks.ceph:Disabling ceph logrotate... 2023-06-23T01:30:44.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T01:30:44.640 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/logrotate.d/ceph 2023-06-23T01:30:44.663 INFO:tasks.ceph:Creating extra log directories... 2023-06-23T01:30:44.664 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T01:30:44.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-06-23T01:30:44.743 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-06-23T01:30:44.744 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-06-23T01:30:44.744 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-06-23T01:30:44.744 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:44.744 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T01:30:44.781 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T01:30:44.895 DEBUG:teuthology.orchestra.run.smithi092:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T01:30:44.920 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:30:44.920 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-06-23T01:30:44.935 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-06-23T01:30:45.048 DEBUG:teuthology.orchestra.run.smithi119:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-06-23T01:30:45.078 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-06-23T01:30:45.079 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-06-23T01:30:45.079 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967', 'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'description': 'fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7311967', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.4', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 9037, 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'fs', 'suite_branch': 'wip-yuri4-testing-2023-06-21-0846-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c', 'targets': {'smithi092.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyCtGo6+Kvz2SWXvHcKq7TwgALZE7O2ZdSSDCKapdOYVDM/+QueBF/SOjjHzoK5xE3qRycvt4MeThoeNOlqGjBDuBnQ1xGPKdvQk852T9BVxz4/2FXr4bBQkf3meLqHxBufyzi07l7SA2ix1Vt6cZKib1QD9uAQL2chWtiTBItmBT16F2fP5qTm46PP+qRVTce+SWbH6QzOXCXiW8Td4zJurxt7VUlWBOVGh8dndG5QnoBLSndUMyiRI+dz34yhUPtRiWXhpDFQXO0Fx3psirsHTyRLczUunFKD3/TpRhcztV4LefNT22EheSHzUazFylz0kl0G9kf4KerisAgZPlJnV9288oXEQDZsgJ3MdZYX9YYcH66DFaveR2jfFYXTBlZ88DEVlQX8o+ThLVFc+K/UwNVlkzBNfsXrkLt+2fYXSgOem9cEuVRue/aG1b4e6Z/Gs2c4UQrwPTk3qA1M8RgQaWMDxq8xtrzs3upVTXBbHJrdjGjbZuVj+/jOeXVyZ8=', 'smithi119.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbatMXeLcMLuQDmdjxHrNuwug4CM7BnUzrhJ0MzSXJQPq5T+21eDfm3gFGTmd8j1sLTQGFLEZdVxC5LMsbJg4z5BLpBBtuwaFKNxM54ePZPOFwXTQx8uUC8aY/xJEecFwDcaIYjlzuQF7zedT+8ZQLWWoUOKmyOqq/VHZIyU0LXNy1yWR3ZSX0Y+L0AjXXlRB85AwxoXlR0mIwXmeT39ozkK/jqz+FDtlnh6mVAf9zpsdl8jKXcMRFKU8uAE4ByOfSiQFkkKIt5FXkGFkc55IvLuodTRi7eB/h08ZeL+ApFKjeOP9WZjNB+mMfLHKBwTBoNl3hYaQZWGPAShxgfeG0tqysLYKVcRfAleAvafOpSWBoaf7JLuhAcZDv1eshFfC98fkIJFlahh0xmzyzwFKXk3zlwmzn+EmOhpUCiYIsksb+6gSz78RsQjAZMFrZukMct8DTaJwqh+gujOG1LEvh4r1jiIvQGokgwAQ9DYQ5ULPE2MqdkpEe9hW7x7npks0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '42ddc46e6e6582d8e29a5a982a81886fe028be2c'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'fail_on_skip': False, 'modules': ['tasks.cephfs.test_volumes.TestVolumes', 'tasks.cephfs.test_volumes.TestSubvolumeGroups', 'tasks.cephfs.test_volumes.TestSubvolumes', 'tasks.cephfs.test_subvolume.TestSubvolume']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '076bbebc42a14f7d568aaa78eabb0038327bcb23', 'timestamp': '2023-06-22_15:25:25', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.23459'} 2023-06-23T01:30:45.080 DEBUG:teuthology.orchestra.run.smithi092:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T01:30:45.082 DEBUG:teuthology.orchestra.run.smithi119:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-06-23T01:30:45.128 DEBUG:teuthology.orchestra.run.smithi092:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T01:30:45.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo install -d -m0777 -- /var/run/ceph 2023-06-23T01:30:45.192 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:45.192 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:30:45.213 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:30:45.214 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_1 2023-06-23T01:30:45.269 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:30:45.269 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:30:45.270 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 213297 Links: 1 2023-06-23T01:30:45.270 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:30:45.270 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:30:45.270 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-06-23 01:30:38.064019011 +0000 2023-06-23T01:30:45.270 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-06-23 01:27:59.759690926 +0000 2023-06-23T01:30:45.270 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-06-23 01:27:59.759690926 +0000 2023-06-23T01:30:45.270 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-06-23T01:30:45.271 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:30:45.331 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-06-23T01:30:45.332 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-06-23T01:30:45.332 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.00026844 s, 1.9 MB/s 2023-06-23T01:30:45.333 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:30:45.389 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_2 2023-06-23T01:30:45.443 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:30:45.444 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:30:45.444 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 211366 Links: 1 2023-06-23T01:30:45.444 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:30:45.444 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:30:45.444 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-06-23 01:30:38.064019011 +0000 2023-06-23T01:30:45.444 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-06-23 01:28:00.275676355 +0000 2023-06-23T01:30:45.445 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-06-23 01:28:00.275676355 +0000 2023-06-23T01:30:45.445 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-06-23T01:30:45.445 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:30:45.507 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-06-23T01:30:45.507 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-06-23T01:30:45.507 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000227114 s, 2.3 MB/s 2023-06-23T01:30:45.508 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:30:45.563 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_3 2023-06-23T01:30:45.618 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:30:45.618 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:30:45.619 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 213425 Links: 1 2023-06-23T01:30:45.619 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:30:45.619 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:30:45.619 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-06-23 01:30:38.065018982 +0000 2023-06-23T01:30:45.619 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-06-23 01:28:00.801661500 +0000 2023-06-23T01:30:45.619 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-06-23 01:28:00.801661500 +0000 2023-06-23T01:30:45.619 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-06-23T01:30:45.620 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:30:45.681 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-06-23T01:30:45.681 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-06-23T01:30:45.681 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000240812 s, 2.1 MB/s 2023-06-23T01:30:45.682 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:30:45.737 DEBUG:teuthology.orchestra.run.smithi092:> stat /dev/vg_nvme/lv_4 2023-06-23T01:30:45.791 INFO:teuthology.orchestra.run.smithi092.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:30:45.792 INFO:teuthology.orchestra.run.smithi092.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:30:45.792 INFO:teuthology.orchestra.run.smithi092.stdout:Device: 6h/6d Inode: 213516 Links: 1 2023-06-23T01:30:45.793 INFO:teuthology.orchestra.run.smithi092.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:30:45.793 INFO:teuthology.orchestra.run.smithi092.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:30:45.793 INFO:teuthology.orchestra.run.smithi092.stdout:Access: 2023-06-23 01:30:38.065018982 +0000 2023-06-23T01:30:45.793 INFO:teuthology.orchestra.run.smithi092.stdout:Modify: 2023-06-23 01:28:01.319646872 +0000 2023-06-23T01:30:45.793 INFO:teuthology.orchestra.run.smithi092.stdout:Change: 2023-06-23 01:28:01.319646872 +0000 2023-06-23T01:30:45.793 INFO:teuthology.orchestra.run.smithi092.stdout: Birth: - 2023-06-23T01:30:45.794 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:30:45.855 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records in 2023-06-23T01:30:45.855 INFO:teuthology.orchestra.run.smithi092.stderr:1+0 records out 2023-06-23T01:30:45.855 INFO:teuthology.orchestra.run.smithi092.stderr:512 bytes copied, 0.000220448 s, 2.3 MB/s 2023-06-23T01:30:45.856 DEBUG:teuthology.orchestra.run.smithi092:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:30:45.911 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:30:45.912 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:30:45.912 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/scratch_devs of=/dev/stdout 2023-06-23T01:30:45.927 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-06-23T01:30:45.927 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_1 2023-06-23T01:30:45.982 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-06-23T01:30:45.982 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:30:45.982 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 211326 Links: 1 2023-06-23T01:30:45.982 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:30:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:30:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2023-06-23 01:30:30.167383002 +0000 2023-06-23T01:30:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2023-06-23 01:27:47.950360603 +0000 2023-06-23T01:30:45.983 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2023-06-23 01:27:47.950360603 +0000 2023-06-23T01:30:45.983 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2023-06-23T01:30:45.983 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-06-23T01:30:46.045 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2023-06-23T01:30:46.046 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2023-06-23T01:30:46.046 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000264128 s, 1.9 MB/s 2023-06-23T01:30:46.047 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-06-23T01:30:46.103 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_2 2023-06-23T01:30:46.157 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-06-23T01:30:46.157 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:30:46.158 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 209375 Links: 1 2023-06-23T01:30:46.158 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:30:46.158 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:30:46.158 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2023-06-23 01:30:30.167383002 +0000 2023-06-23T01:30:46.158 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2023-06-23 01:27:48.459348166 +0000 2023-06-23T01:30:46.158 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2023-06-23 01:27:48.459348166 +0000 2023-06-23T01:30:46.158 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2023-06-23T01:30:46.159 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-06-23T01:30:46.219 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2023-06-23T01:30:46.219 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2023-06-23T01:30:46.220 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000249528 s, 2.1 MB/s 2023-06-23T01:30:46.220 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-06-23T01:30:46.276 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_3 2023-06-23T01:30:46.330 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-06-23T01:30:46.331 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:30:46.331 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 211446 Links: 1 2023-06-23T01:30:46.331 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:30:46.331 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:30:46.331 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2023-06-23 01:30:30.167383002 +0000 2023-06-23T01:30:46.331 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2023-06-23 01:27:48.976335537 +0000 2023-06-23T01:30:46.332 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2023-06-23 01:27:48.976335537 +0000 2023-06-23T01:30:46.332 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2023-06-23T01:30:46.332 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-06-23T01:30:46.393 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2023-06-23T01:30:46.394 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2023-06-23T01:30:46.394 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000241783 s, 2.1 MB/s 2023-06-23T01:30:46.395 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-06-23T01:30:46.450 DEBUG:teuthology.orchestra.run.smithi119:> stat /dev/vg_nvme/lv_4 2023-06-23T01:30:46.504 INFO:teuthology.orchestra.run.smithi119.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-06-23T01:30:46.505 INFO:teuthology.orchestra.run.smithi119.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-06-23T01:30:46.505 INFO:teuthology.orchestra.run.smithi119.stdout:Device: 6h/6d Inode: 212376 Links: 1 2023-06-23T01:30:46.505 INFO:teuthology.orchestra.run.smithi119.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-06-23T01:30:46.505 INFO:teuthology.orchestra.run.smithi119.stdout:Context: system_u:object_r:device_t:s0 2023-06-23T01:30:46.505 INFO:teuthology.orchestra.run.smithi119.stdout:Access: 2023-06-23 01:30:30.167383002 +0000 2023-06-23T01:30:46.505 INFO:teuthology.orchestra.run.smithi119.stdout:Modify: 2023-06-23 01:27:49.493322906 +0000 2023-06-23T01:30:46.505 INFO:teuthology.orchestra.run.smithi119.stdout:Change: 2023-06-23 01:27:49.493322906 +0000 2023-06-23T01:30:46.506 INFO:teuthology.orchestra.run.smithi119.stdout: Birth: - 2023-06-23T01:30:46.506 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-06-23T01:30:46.568 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records in 2023-06-23T01:30:46.569 INFO:teuthology.orchestra.run.smithi119.stderr:1+0 records out 2023-06-23T01:30:46.569 INFO:teuthology.orchestra.run.smithi119.stderr:512 bytes copied, 0.000197548 s, 2.6 MB/s 2023-06-23T01:30:46.570 DEBUG:teuthology.orchestra.run.smithi119:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-06-23T01:30:46.624 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:30:46.625 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T01:30:46.625 INFO:tasks.ceph:Generating config... 2023-06-23T01:30:46.627 INFO:tasks.ceph:[client] client mount timeout = 600 2023-06-23T01:30:46.627 INFO:tasks.ceph:[client] debug client = 20 2023-06-23T01:30:46.627 INFO:tasks.ceph:[client] debug ms = 1 2023-06-23T01:30:46.628 INFO:tasks.ceph:[client] pid file = 2023-06-23T01:30:46.628 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-06-23T01:30:46.628 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-06-23T01:30:46.629 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-06-23T01:30:46.629 INFO:tasks.ceph:[mds] debug mds = 20 2023-06-23T01:30:46.629 INFO:tasks.ceph:[mds] debug ms = 1 2023-06-23T01:30:46.629 INFO:tasks.ceph:[mds] mds debug frag = True 2023-06-23T01:30:46.629 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-06-23T01:30:46.630 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-06-23T01:30:46.630 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-06-23T01:30:46.630 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-06-23T01:30:46.631 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-06-23T01:30:46.631 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-06-23T01:30:46.631 INFO:tasks.ceph:[mgr] debug client = 20 2023-06-23T01:30:46.631 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-06-23T01:30:46.631 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-06-23T01:30:46.632 INFO:tasks.ceph:[mgr] debug ms = 1 2023-06-23T01:30:46.632 INFO:tasks.ceph:[mon] debug mon = 20 2023-06-23T01:30:46.632 INFO:tasks.ceph:[mon] debug ms = 1 2023-06-23T01:30:46.632 INFO:tasks.ceph:[mon] debug paxos = 20 2023-06-23T01:30:46.632 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-06-23T01:30:46.633 INFO:tasks.ceph:[osd] bdev async discard = True 2023-06-23T01:30:46.633 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-06-23T01:30:46.633 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-06-23T01:30:46.633 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-06-23T01:30:46.633 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-06-23T01:30:46.634 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-06-23T01:30:46.634 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-06-23T01:30:46.634 INFO:tasks.ceph:[osd] debug ms = 1 2023-06-23T01:30:46.634 INFO:tasks.ceph:[osd] debug osd = 20 2023-06-23T01:30:46.635 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-06-23T01:30:46.635 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-06-23T01:30:46.635 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-06-23T01:30:46.635 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-06-23T01:30:46.636 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-06-23T01:30:46.636 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-06-23T01:30:46.636 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-06-23T01:30:46.636 INFO:tasks.ceph:Setting up mon.a... 2023-06-23T01:30:46.637 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-06-23T01:30:46.685 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.keyring 2023-06-23T01:30:46.688 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-06-23T01:30:46.728 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:30:46.793 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.92'), ('mon.b', '172.21.15.119'), ('mon.c', '[v2:172.21.15.119:3301,v1:172.21.15.119:6790]')] 2023-06-23T01:30:46.794 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.92,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-06-23T01:30:46.795 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:46.796 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T01:30:46.848 DEBUG:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.92 --add b 172.21.15.119 --addv c '[v2:172.21.15.119:3301,v1:172.21.15.119:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:30:46.931 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:30:46.931 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: generated fsid c94811cf-f7e5-4c57-b505-a983532f8da8 2023-06-23T01:30:46.931 INFO:teuthology.orchestra.run.smithi092.stdout:setting min_mon_release = octopus 2023-06-23T01:30:46.931 INFO:teuthology.orchestra.run.smithi092.stdout:epoch 0 2023-06-23T01:30:46.932 INFO:teuthology.orchestra.run.smithi092.stdout:fsid c94811cf-f7e5-4c57-b505-a983532f8da8 2023-06-23T01:30:46.932 INFO:teuthology.orchestra.run.smithi092.stdout:last_changed 2023-06-23T01:30:46.930377+0000 2023-06-23T01:30:46.932 INFO:teuthology.orchestra.run.smithi092.stdout:created 2023-06-23T01:30:46.930377+0000 2023-06-23T01:30:46.932 INFO:teuthology.orchestra.run.smithi092.stdout:min_mon_release 15 (octopus) 2023-06-23T01:30:46.933 INFO:teuthology.orchestra.run.smithi092.stdout:election_strategy: 1 2023-06-23T01:30:46.933 INFO:teuthology.orchestra.run.smithi092.stdout:0: [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] mon.a 2023-06-23T01:30:46.933 INFO:teuthology.orchestra.run.smithi092.stdout:1: [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] mon.b 2023-06-23T01:30:46.933 INFO:teuthology.orchestra.run.smithi092.stdout:2: [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] mon.c 2023-06-23T01:30:46.934 INFO:teuthology.orchestra.run.smithi092.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-06-23T01:30:46.936 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-06-23T01:30:46.989 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID c94811cf-f7e5-4c57-b505-a983532f8da8... 2023-06-23T01:30:46.991 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T01:30:47.031 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-06-23T01:30:47.072 INFO:teuthology.orchestra.run.smithi092.stdout:[global] 2023-06-23T01:30:47.072 INFO:teuthology.orchestra.run.smithi092.stdout: chdir = "" 2023-06-23T01:30:47.072 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T01:30:47.072 INFO:teuthology.orchestra.run.smithi092.stdout: auth supported = cephx 2023-06-23T01:30:47.072 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.073 INFO:teuthology.orchestra.run.smithi092.stdout: filestore xattr use omap = true 2023-06-23T01:30:47.073 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.073 INFO:teuthology.orchestra.run.smithi092.stdout: mon clock drift allowed = 1.000 2023-06-23T01:30:47.073 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.073 INFO:teuthology.orchestra.run.smithi092.stdout: osd crush chooseleaf type = 0 2023-06-23T01:30:47.073 INFO:teuthology.orchestra.run.smithi092.stdout: auth debug = true 2023-06-23T01:30:47.074 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.074 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on old message = true 2023-06-23T01:30:47.074 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on bug = true 2023-06-23T01:30:47.074 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.074 INFO:teuthology.orchestra.run.smithi092.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T01:30:47.074 INFO:teuthology.orchestra.run.smithi092.stdout: mon pg warn max object skew = 0 2023-06-23T01:30:47.074 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.075 INFO:teuthology.orchestra.run.smithi092.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T01:30:47.075 INFO:teuthology.orchestra.run.smithi092.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T01:30:47.075 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.075 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default size = 2 2023-06-23T01:30:47.075 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow primary affinity = true 2023-06-23T01:30:47.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd allow pg remap = true 2023-06-23T01:30:47.075 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on legacy crush tunables = false 2023-06-23T01:30:47.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on crush straw calc version zero = false 2023-06-23T01:30:47.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on no sortbitwise = false 2023-06-23T01:30:47.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on osd down out interval zero = false 2023-06-23T01:30:47.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon warn on too few osds = false 2023-06-23T01:30:47.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T01:30:47.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T01:30:47.076 INFO:teuthology.orchestra.run.smithi092.stdout: mon_allow_pool_size_one = true 2023-06-23T01:30:47.077 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T01:30:47.077 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.077 INFO:teuthology.orchestra.run.smithi092.stdout: osd default data pool replay window = 5 2023-06-23T01:30:47.077 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.077 INFO:teuthology.orchestra.run.smithi092.stdout: mon allow pool delete = true 2023-06-23T01:30:47.077 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.077 INFO:teuthology.orchestra.run.smithi092.stdout: mon cluster log file level = debug 2023-06-23T01:30:47.078 INFO:teuthology.orchestra.run.smithi092.stdout: debug asserts on shutdown = true 2023-06-23T01:30:47.078 INFO:teuthology.orchestra.run.smithi092.stdout: mon health detail to clog = false 2023-06-23T01:30:47.078 INFO:teuthology.orchestra.run.smithi092.stdout: mon host = "172.21.15.92,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]" 2023-06-23T01:30:47.078 INFO:teuthology.orchestra.run.smithi092.stdout: ms die on skipped message = False 2023-06-23T01:30:47.078 INFO:teuthology.orchestra.run.smithi092.stdout: fsid = c94811cf-f7e5-4c57-b505-a983532f8da8 2023-06-23T01:30:47.078 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.078 INFO:teuthology.orchestra.run.smithi092.stdout:[osd] 2023-06-23T01:30:47.078 INFO:teuthology.orchestra.run.smithi092.stdout: osd journal size = 100 2023-06-23T01:30:47.079 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.079 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub load threshold = 5.0 2023-06-23T01:30:47.079 INFO:teuthology.orchestra.run.smithi092.stdout: osd scrub max interval = 600 2023-06-23T01:30:47.079 INFO:teuthology.orchestra.run.smithi092.stdout: osd mclock profile = high_recovery_ops 2023-06-23T01:30:47.079 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.079 INFO:teuthology.orchestra.run.smithi092.stdout: osd recover clone overlap = true 2023-06-23T01:30:47.079 INFO:teuthology.orchestra.run.smithi092.stdout: osd recovery max chunk = 1048576 2023-06-23T01:30:47.079 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug shutdown = true 2023-06-23T01:30:47.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug op order = true 2023-06-23T01:30:47.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug verify stray on activate = true 2023-06-23T01:30:47.080 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd open classes on start = true 2023-06-23T01:30:47.080 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug pg log writeout = true 2023-06-23T01:30:47.080 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.081 INFO:teuthology.orchestra.run.smithi092.stdout: osd deep scrub update digest min age = 30 2023-06-23T01:30:47.081 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.081 INFO:teuthology.orchestra.run.smithi092.stdout: osd map max advance = 10 2023-06-23T01:30:47.081 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.081 INFO:teuthology.orchestra.run.smithi092.stdout: journal zero on create = true 2023-06-23T01:30:47.081 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.081 INFO:teuthology.orchestra.run.smithi092.stdout: filestore ondisk finisher threads = 3 2023-06-23T01:30:47.081 INFO:teuthology.orchestra.run.smithi092.stdout: filestore apply finisher threads = 3 2023-06-23T01:30:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: bdev debug aio = true 2023-06-23T01:30:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: osd debug misdirected ops = true 2023-06-23T01:30:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: bdev async discard = True 2023-06-23T01:30:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: bdev enable discard = True 2023-06-23T01:30:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore allocator = bitmap 2023-06-23T01:30:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore block size = 96636764160 2023-06-23T01:30:47.082 INFO:teuthology.orchestra.run.smithi092.stdout: bluestore fsck on mount = True 2023-06-23T01:30:47.083 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluefs = 1/20 2023-06-23T01:30:47.083 INFO:teuthology.orchestra.run.smithi092.stdout: debug bluestore = 1/20 2023-06-23T01:30:47.083 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-06-23T01:30:47.083 INFO:teuthology.orchestra.run.smithi092.stdout: debug osd = 20 2023-06-23T01:30:47.083 INFO:teuthology.orchestra.run.smithi092.stdout: debug rocksdb = 4/10 2023-06-23T01:30:47.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T01:30:47.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd full ratio = 0.9 2023-06-23T01:30:47.083 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd nearfull ratio = 0.8 2023-06-23T01:30:47.084 INFO:teuthology.orchestra.run.smithi092.stdout: osd failsafe full ratio = 0.95 2023-06-23T01:30:47.084 INFO:teuthology.orchestra.run.smithi092.stdout: osd objectstore = bluestore 2023-06-23T01:30:47.084 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2023-06-23T01:30:47.084 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.084 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr] 2023-06-23T01:30:47.085 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-06-23T01:30:47.085 INFO:teuthology.orchestra.run.smithi092.stdout: debug mgr = 20 2023-06-23T01:30:47.085 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2023-06-23T01:30:47.085 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2023-06-23T01:30:47.085 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:30:47.085 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:30:47.085 INFO:teuthology.orchestra.run.smithi092.stdout: mgr/telemetry/nag = false 2023-06-23T01:30:47.085 INFO:teuthology.orchestra.run.smithi092.stdout: debug client = 20 2023-06-23T01:30:47.086 INFO:teuthology.orchestra.run.smithi092.stdout: debug finisher = 20 2023-06-23T01:30:47.086 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.086 INFO:teuthology.orchestra.run.smithi092.stdout:[mon] 2023-06-23T01:30:47.086 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-06-23T01:30:47.086 INFO:teuthology.orchestra.run.smithi092.stdout: debug mon = 20 2023-06-23T01:30:47.086 INFO:teuthology.orchestra.run.smithi092.stdout: debug paxos = 20 2023-06-23T01:30:47.086 INFO:teuthology.orchestra.run.smithi092.stdout: debug auth = 20 2023-06-23T01:30:47.086 INFO:teuthology.orchestra.run.smithi092.stdout: mon data avail warn = 5 2023-06-23T01:30:47.087 INFO:teuthology.orchestra.run.smithi092.stdout: mon mgr mkfs grace = 240 2023-06-23T01:30:47.087 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:30:47.087 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd reporter subtree level = osd 2023-06-23T01:30:47.087 INFO:teuthology.orchestra.run.smithi092.stdout: mon osd prime pg temp = true 2023-06-23T01:30:47.087 INFO:teuthology.orchestra.run.smithi092.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:30:47.087 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.087 INFO:teuthology.orchestra.run.smithi092.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T01:30:47.088 INFO:teuthology.orchestra.run.smithi092.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T01:30:47.088 INFO:teuthology.orchestra.run.smithi092.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T01:30:47.088 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.088 INFO:teuthology.orchestra.run.smithi092.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T01:30:47.088 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T01:30:47.088 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T01:30:47.088 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.089 INFO:teuthology.orchestra.run.smithi092.stdout: # 1m isn't quite enough 2023-06-23T01:30:47.089 INFO:teuthology.orchestra.run.smithi092.stdout: mon_down_mkfs_grace = 2m 2023-06-23T01:30:47.089 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.089 INFO:teuthology.orchestra.run.smithi092.stdout: mon_warn_on_filestore_osds = false 2023-06-23T01:30:47.089 INFO:teuthology.orchestra.run.smithi092.stdout: mon op complaint time = 120 2023-06-23T01:30:47.089 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:30:47.089 INFO:teuthology.orchestra.run.smithi092.stdout:[client] 2023-06-23T01:30:47.090 INFO:teuthology.orchestra.run.smithi092.stdout: rgw cache enabled = true 2023-06-23T01:30:47.090 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable ops log = true 2023-06-23T01:30:47.090 INFO:teuthology.orchestra.run.smithi092.stdout: rgw enable usage log = true 2023-06-23T01:30:47.090 INFO:teuthology.orchestra.run.smithi092.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T01:30:47.090 INFO:teuthology.orchestra.run.smithi092.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T01:30:47.090 INFO:teuthology.orchestra.run.smithi092.stdout: client mount timeout = 600 2023-06-23T01:30:47.090 INFO:teuthology.orchestra.run.smithi092.stdout: debug client = 20 2023-06-23T01:30:47.090 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-06-23T01:30:47.091 INFO:teuthology.orchestra.run.smithi092.stdout: pid file = "" 2023-06-23T01:30:47.091 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 900 2023-06-23T01:30:47.091 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 900 2023-06-23T01:30:47.091 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.a] 2023-06-23T01:30:47.091 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.b] 2023-06-23T01:30:47.091 INFO:teuthology.orchestra.run.smithi092.stdout:[mon.c] 2023-06-23T01:30:47.091 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2023-06-23T01:30:47.091 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.c] 2023-06-23T01:30:47.092 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2023-06-23T01:30:47.092 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.d] 2023-06-23T01:30:47.092 INFO:teuthology.orchestra.run.smithi092.stdout:[mds] 2023-06-23T01:30:47.092 INFO:teuthology.orchestra.run.smithi092.stdout: debug mds = 20 2023-06-23T01:30:47.092 INFO:teuthology.orchestra.run.smithi092.stdout: debug ms = 1 2023-06-23T01:30:47.092 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug frag = True 2023-06-23T01:30:47.092 INFO:teuthology.orchestra.run.smithi092.stdout: mds debug scatterstat = True 2023-06-23T01:30:47.093 INFO:teuthology.orchestra.run.smithi092.stdout: mds op complaint time = 180 2023-06-23T01:30:47.093 INFO:teuthology.orchestra.run.smithi092.stdout: mds verify scatter = True 2023-06-23T01:30:47.093 INFO:teuthology.orchestra.run.smithi092.stdout: osd op complaint time = 180 2023-06-23T01:30:47.093 INFO:teuthology.orchestra.run.smithi092.stdout: rados mon op timeout = 900 2023-06-23T01:30:47.093 INFO:teuthology.orchestra.run.smithi092.stdout: rados osd op timeout = 900 2023-06-23T01:30:47.094 INFO:teuthology.orchestra.run.smithi119.stdout:[global] 2023-06-23T01:30:47.095 INFO:teuthology.orchestra.run.smithi119.stdout: chdir = "" 2023-06-23T01:30:47.095 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-06-23T01:30:47.095 INFO:teuthology.orchestra.run.smithi119.stdout: auth supported = cephx 2023-06-23T01:30:47.095 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.095 INFO:teuthology.orchestra.run.smithi119.stdout: filestore xattr use omap = true 2023-06-23T01:30:47.095 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.096 INFO:teuthology.orchestra.run.smithi119.stdout: mon clock drift allowed = 1.000 2023-06-23T01:30:47.096 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.096 INFO:teuthology.orchestra.run.smithi119.stdout: osd crush chooseleaf type = 0 2023-06-23T01:30:47.096 INFO:teuthology.orchestra.run.smithi119.stdout: auth debug = true 2023-06-23T01:30:47.096 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.096 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on old message = true 2023-06-23T01:30:47.096 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on bug = true 2023-06-23T01:30:47.097 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.097 INFO:teuthology.orchestra.run.smithi119.stdout: mon max pg per osd = 10000 # >= luminous 2023-06-23T01:30:47.097 INFO:teuthology.orchestra.run.smithi119.stdout: mon pg warn max object skew = 0 2023-06-23T01:30:47.097 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.097 INFO:teuthology.orchestra.run.smithi119.stdout: # disable pg_autoscaler by default for new pools 2023-06-23T01:30:47.097 INFO:teuthology.orchestra.run.smithi119.stdout: osd_pool_default_pg_autoscale_mode = off 2023-06-23T01:30:47.097 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.097 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default size = 2 2023-06-23T01:30:47.098 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.098 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow primary affinity = true 2023-06-23T01:30:47.098 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd allow pg remap = true 2023-06-23T01:30:47.098 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on legacy crush tunables = false 2023-06-23T01:30:47.098 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on crush straw calc version zero = false 2023-06-23T01:30:47.098 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on no sortbitwise = false 2023-06-23T01:30:47.098 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on osd down out interval zero = false 2023-06-23T01:30:47.099 INFO:teuthology.orchestra.run.smithi119.stdout: mon warn on too few osds = false 2023-06-23T01:30:47.099 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-06-23T01:30:47.099 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_pool_no_redundancy = false 2023-06-23T01:30:47.099 INFO:teuthology.orchestra.run.smithi119.stdout: mon_allow_pool_size_one = true 2023-06-23T01:30:47.099 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.099 INFO:teuthology.orchestra.run.smithi119.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-06-23T01:30:47.099 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.099 INFO:teuthology.orchestra.run.smithi119.stdout: osd default data pool replay window = 5 2023-06-23T01:30:47.100 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.100 INFO:teuthology.orchestra.run.smithi119.stdout: mon allow pool delete = true 2023-06-23T01:30:47.100 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.100 INFO:teuthology.orchestra.run.smithi119.stdout: mon cluster log file level = debug 2023-06-23T01:30:47.100 INFO:teuthology.orchestra.run.smithi119.stdout: debug asserts on shutdown = true 2023-06-23T01:30:47.100 INFO:teuthology.orchestra.run.smithi119.stdout: mon health detail to clog = false 2023-06-23T01:30:47.100 INFO:teuthology.orchestra.run.smithi119.stdout: mon host = "172.21.15.92,172.21.15.119,[v2:172.21.15.119:3301,v1:172.21.15.119:6790]" 2023-06-23T01:30:47.101 INFO:teuthology.orchestra.run.smithi119.stdout: ms die on skipped message = False 2023-06-23T01:30:47.101 INFO:teuthology.orchestra.run.smithi119.stdout: fsid = c94811cf-f7e5-4c57-b505-a983532f8da8 2023-06-23T01:30:47.101 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.101 INFO:teuthology.orchestra.run.smithi119.stdout:[osd] 2023-06-23T01:30:47.101 INFO:teuthology.orchestra.run.smithi119.stdout: osd journal size = 100 2023-06-23T01:30:47.101 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.101 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub load threshold = 5.0 2023-06-23T01:30:47.101 INFO:teuthology.orchestra.run.smithi119.stdout: osd scrub max interval = 600 2023-06-23T01:30:47.102 INFO:teuthology.orchestra.run.smithi119.stdout: osd mclock profile = high_recovery_ops 2023-06-23T01:30:47.102 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.102 INFO:teuthology.orchestra.run.smithi119.stdout: osd recover clone overlap = true 2023-06-23T01:30:47.102 INFO:teuthology.orchestra.run.smithi119.stdout: osd recovery max chunk = 1048576 2023-06-23T01:30:47.102 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.102 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug shutdown = true 2023-06-23T01:30:47.102 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug op order = true 2023-06-23T01:30:47.102 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug verify stray on activate = true 2023-06-23T01:30:47.103 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.103 INFO:teuthology.orchestra.run.smithi119.stdout: osd open classes on start = true 2023-06-23T01:30:47.103 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug pg log writeout = true 2023-06-23T01:30:47.103 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.103 INFO:teuthology.orchestra.run.smithi119.stdout: osd deep scrub update digest min age = 30 2023-06-23T01:30:47.103 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.103 INFO:teuthology.orchestra.run.smithi119.stdout: osd map max advance = 10 2023-06-23T01:30:47.103 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.104 INFO:teuthology.orchestra.run.smithi119.stdout: journal zero on create = true 2023-06-23T01:30:47.104 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.104 INFO:teuthology.orchestra.run.smithi119.stdout: filestore ondisk finisher threads = 3 2023-06-23T01:30:47.104 INFO:teuthology.orchestra.run.smithi119.stdout: filestore apply finisher threads = 3 2023-06-23T01:30:47.104 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.104 INFO:teuthology.orchestra.run.smithi119.stdout: bdev debug aio = true 2023-06-23T01:30:47.104 INFO:teuthology.orchestra.run.smithi119.stdout: osd debug misdirected ops = true 2023-06-23T01:30:47.105 INFO:teuthology.orchestra.run.smithi119.stdout: bdev async discard = True 2023-06-23T01:30:47.105 INFO:teuthology.orchestra.run.smithi119.stdout: bdev enable discard = True 2023-06-23T01:30:47.105 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore allocator = bitmap 2023-06-23T01:30:47.105 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore block size = 96636764160 2023-06-23T01:30:47.105 INFO:teuthology.orchestra.run.smithi119.stdout: bluestore fsck on mount = True 2023-06-23T01:30:47.105 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluefs = 1/20 2023-06-23T01:30:47.106 INFO:teuthology.orchestra.run.smithi119.stdout: debug bluestore = 1/20 2023-06-23T01:30:47.106 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2023-06-23T01:30:47.106 INFO:teuthology.orchestra.run.smithi119.stdout: debug osd = 20 2023-06-23T01:30:47.106 INFO:teuthology.orchestra.run.smithi119.stdout: debug rocksdb = 4/10 2023-06-23T01:30:47.106 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd backfillfull_ratio = 0.85 2023-06-23T01:30:47.106 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd full ratio = 0.9 2023-06-23T01:30:47.106 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd nearfull ratio = 0.8 2023-06-23T01:30:47.106 INFO:teuthology.orchestra.run.smithi119.stdout: osd failsafe full ratio = 0.95 2023-06-23T01:30:47.107 INFO:teuthology.orchestra.run.smithi119.stdout: osd objectstore = bluestore 2023-06-23T01:30:47.107 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2023-06-23T01:30:47.107 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.107 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr] 2023-06-23T01:30:47.107 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2023-06-23T01:30:47.107 INFO:teuthology.orchestra.run.smithi119.stdout: debug mgr = 20 2023-06-23T01:30:47.107 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2023-06-23T01:30:47.108 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2023-06-23T01:30:47.108 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:30:47.108 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:30:47.108 INFO:teuthology.orchestra.run.smithi119.stdout: mgr/telemetry/nag = false 2023-06-23T01:30:47.108 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2023-06-23T01:30:47.108 INFO:teuthology.orchestra.run.smithi119.stdout: debug finisher = 20 2023-06-23T01:30:47.108 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.109 INFO:teuthology.orchestra.run.smithi119.stdout:[mon] 2023-06-23T01:30:47.109 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2023-06-23T01:30:47.109 INFO:teuthology.orchestra.run.smithi119.stdout: debug mon = 20 2023-06-23T01:30:47.109 INFO:teuthology.orchestra.run.smithi119.stdout: debug paxos = 20 2023-06-23T01:30:47.109 INFO:teuthology.orchestra.run.smithi119.stdout: debug auth = 20 2023-06-23T01:30:47.109 INFO:teuthology.orchestra.run.smithi119.stdout: mon data avail warn = 5 2023-06-23T01:30:47.109 INFO:teuthology.orchestra.run.smithi119.stdout: mon mgr mkfs grace = 240 2023-06-23T01:30:47.109 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min pgs per osd = 4 2023-06-23T01:30:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd reporter subtree level = osd 2023-06-23T01:30:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: mon osd prime pg temp = true 2023-06-23T01:30:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: mon reweight min bytes per osd = 10 2023-06-23T01:30:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-06-23T01:30:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: auth mon ticket ttl = 660 # 11m 2023-06-23T01:30:47.110 INFO:teuthology.orchestra.run.smithi119.stdout: auth service ticket ttl = 240 # 4m 2023-06-23T01:30:47.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.111 INFO:teuthology.orchestra.run.smithi119.stdout: # don't complain about insecure global_id in the test suite 2023-06-23T01:30:47.111 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-06-23T01:30:47.111 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-06-23T01:30:47.111 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.111 INFO:teuthology.orchestra.run.smithi119.stdout: # 1m isn't quite enough 2023-06-23T01:30:47.111 INFO:teuthology.orchestra.run.smithi119.stdout: mon_down_mkfs_grace = 2m 2023-06-23T01:30:47.112 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.112 INFO:teuthology.orchestra.run.smithi119.stdout: mon_warn_on_filestore_osds = false 2023-06-23T01:30:47.112 INFO:teuthology.orchestra.run.smithi119.stdout: mon op complaint time = 120 2023-06-23T01:30:47.112 INFO:teuthology.orchestra.run.smithi119.stdout: 2023-06-23T01:30:47.112 INFO:teuthology.orchestra.run.smithi119.stdout:[client] 2023-06-23T01:30:47.112 INFO:teuthology.orchestra.run.smithi119.stdout: rgw cache enabled = true 2023-06-23T01:30:47.112 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable ops log = true 2023-06-23T01:30:47.112 INFO:teuthology.orchestra.run.smithi119.stdout: rgw enable usage log = true 2023-06-23T01:30:47.113 INFO:teuthology.orchestra.run.smithi119.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-06-23T01:30:47.113 INFO:teuthology.orchestra.run.smithi119.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-06-23T01:30:47.113 INFO:teuthology.orchestra.run.smithi119.stdout: client mount timeout = 600 2023-06-23T01:30:47.113 INFO:teuthology.orchestra.run.smithi119.stdout: debug client = 20 2023-06-23T01:30:47.113 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2023-06-23T01:30:47.113 INFO:teuthology.orchestra.run.smithi119.stdout: pid file = "" 2023-06-23T01:30:47.114 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2023-06-23T01:30:47.114 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2023-06-23T01:30:47.114 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.a] 2023-06-23T01:30:47.115 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.b] 2023-06-23T01:30:47.115 INFO:teuthology.orchestra.run.smithi119.stdout:[mon.c] 2023-06-23T01:30:47.115 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2023-06-23T01:30:47.115 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2023-06-23T01:30:47.115 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2023-06-23T01:30:47.115 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2023-06-23T01:30:47.115 INFO:teuthology.orchestra.run.smithi119.stdout:[mds] 2023-06-23T01:30:47.116 INFO:teuthology.orchestra.run.smithi119.stdout: debug mds = 20 2023-06-23T01:30:47.116 INFO:teuthology.orchestra.run.smithi119.stdout: debug ms = 1 2023-06-23T01:30:47.116 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug frag = True 2023-06-23T01:30:47.116 INFO:teuthology.orchestra.run.smithi119.stdout: mds debug scatterstat = True 2023-06-23T01:30:47.116 INFO:teuthology.orchestra.run.smithi119.stdout: mds op complaint time = 180 2023-06-23T01:30:47.116 INFO:teuthology.orchestra.run.smithi119.stdout: mds verify scatter = True 2023-06-23T01:30:47.116 INFO:teuthology.orchestra.run.smithi119.stdout: osd op complaint time = 180 2023-06-23T01:30:47.117 INFO:teuthology.orchestra.run.smithi119.stdout: rados mon op timeout = 900 2023-06-23T01:30:47.117 INFO:teuthology.orchestra.run.smithi119.stdout: rados osd op timeout = 900 2023-06-23T01:30:47.117 INFO:tasks.ceph:Creating admin key on mon.a... 2023-06-23T01:30:47.117 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-06-23T01:30:47.185 INFO:tasks.ceph:Copying monmap to all nodes... 2023-06-23T01:30:47.186 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:47.186 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-06-23T01:30:47.200 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:47.201 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-06-23T01:30:47.254 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:30:47.254 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:47.254 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T01:30:47.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:30:47.328 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:30:47.328 DEBUG:teuthology.orchestra.run.smithi092:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:30:47.382 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:30:47.382 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:30:47.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd of=/etc/ceph/ceph.keyring 2023-06-23T01:30:47.383 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-06-23T01:30:47.417 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:30:47.417 DEBUG:teuthology.orchestra.run.smithi119:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:30:47.472 INFO:tasks.ceph:Setting up mon nodes... 2023-06-23T01:30:47.472 INFO:tasks.ceph:Setting up mgr nodes... 2023-06-23T01:30:47.472 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T01:30:47.519 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-06-23T01:30:47.521 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T01:30:47.596 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-06-23T01:30:47.599 INFO:tasks.ceph:Setting up mds nodes... 2023-06-23T01:30:47.599 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T01:30:47.647 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-06-23T01:30:47.649 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-06-23T01:30:47.673 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T01:30:47.760 INFO:teuthology.orchestra.run.smithi092.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-06-23T01:30:47.762 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-06-23T01:30:47.786 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T01:30:47.833 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-06-23T01:30:47.836 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-06-23T01:30:47.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T01:30:47.946 INFO:teuthology.orchestra.run.smithi119.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-06-23T01:30:47.948 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-06-23T01:30:47.972 INFO:tasks.ceph_client:Setting up client nodes... 2023-06-23T01:30:47.973 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-06-23T01:30:48.011 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-06-23T01:30:48.023 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-06-23T01:30:48.100 INFO:teuthology.orchestra.run.smithi092.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-06-23T01:30:48.113 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-06-23T01:30:48.151 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-06-23T01:30:48.163 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-06-23T01:30:48.242 INFO:teuthology.orchestra.run.smithi119.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-06-23T01:30:48.254 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-06-23T01:30:48.255 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi092.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi119.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-06-23T01:30:48.255 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-06-23T01:30:48.282 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:30:48.283 INFO:tasks.ceph:role: osd.0 2023-06-23T01:30:48.283 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:30:48.283 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T01:30:48.353 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:30:48.354 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:30:48.354 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:30:48.354 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 2023-06-23T01:30:48.354 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:30:48.354 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:30:48.354 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:30:48.354 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:30:48.355 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:30:48.355 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:30:48.779 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-06-23T01:30:48.781 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-06-23T01:30:48.782 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-06-23T01:30:48.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-06-23T01:30:48.986 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-06-23T01:30:49.050 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:30:49.051 INFO:tasks.ceph:role: osd.1 2023-06-23T01:30:49.051 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:30:49.051 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T01:30:49.122 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:30:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:30:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:30:49.123 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 2023-06-23T01:30:49.124 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:30:49.124 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:30:49.124 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:30:49.124 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:30:49.125 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:30:49.125 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:30:49.550 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-06-23T01:30:49.552 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-06-23T01:30:49.552 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-06-23T01:30:49.662 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-06-23T01:30:49.690 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-06-23T01:30:49.754 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:30:49.755 INFO:tasks.ceph:role: osd.2 2023-06-23T01:30:49.756 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:30:49.756 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T01:30:49.827 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:30:49.827 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:30:49.827 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:30:49.827 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 2023-06-23T01:30:49.828 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:30:49.828 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:30:49.828 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:30:49.828 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:30:49.828 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:30:49.828 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:30:50.274 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-06-23T01:30:50.276 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-06-23T01:30:50.277 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-06-23T01:30:50.446 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-06-23T01:30:50.477 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-06-23T01:30:50.541 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-06-23T01:30:50.542 INFO:tasks.ceph:role: osd.3 2023-06-23T01:30:50.542 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:30:50.542 DEBUG:teuthology.orchestra.run.smithi092:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T01:30:50.614 INFO:teuthology.orchestra.run.smithi092.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:30:50.614 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:30:50.614 INFO:teuthology.orchestra.run.smithi092.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:30:50.614 INFO:teuthology.orchestra.run.smithi092.stdout: = reflink=1 2023-06-23T01:30:50.614 INFO:teuthology.orchestra.run.smithi092.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:30:50.614 INFO:teuthology.orchestra.run.smithi092.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:30:50.615 INFO:teuthology.orchestra.run.smithi092.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:30:50.615 INFO:teuthology.orchestra.run.smithi092.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:30:50.615 INFO:teuthology.orchestra.run.smithi092.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:30:50.615 INFO:teuthology.orchestra.run.smithi092.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:30:51.056 INFO:teuthology.orchestra.run.smithi092.stdout:Discarding blocks...Done. 2023-06-23T01:30:51.061 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi092.front.sepia.ceph.com -o noatime 2023-06-23T01:30:51.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-06-23T01:30:51.232 DEBUG:teuthology.orchestra.run.smithi092:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-06-23T01:30:51.263 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:30:51.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:51.360+0000 7f3809933540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-06-23T01:30:51.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:51.360+0000 7f3809933540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-06-23T01:30:51.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:51.360+0000 7f3809933540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:30:51.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:51.360+0000 7f3809933540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:30:51.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:51.360+0000 7f3809933540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-06-23T01:30:51.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:51.361+0000 7f3809933540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-06-23T01:30:51.885 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:51.882+0000 7f3809933540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:30:52.649 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:52.647+0000 7f3809933540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:30:53.413 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:53.411+0000 7f3809933540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:30:54.205 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-06-23T01:30:54.234 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:30:54.322 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:54.320+0000 7f1f50dd4540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-06-23T01:30:54.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:54.320+0000 7f1f50dd4540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-06-23T01:30:54.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:54.320+0000 7f1f50dd4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:30:54.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:54.320+0000 7f1f50dd4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:30:54.323 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:54.320+0000 7f1f50dd4540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-06-23T01:30:54.324 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:54.321+0000 7f1f50dd4540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-06-23T01:30:54.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:54.843+0000 7f1f50dd4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:30:55.607 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:55.604+0000 7f1f50dd4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:30:56.369 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:56.366+0000 7f1f50dd4540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:30:57.169 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-06-23T01:30:57.199 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:30:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:57.285+0000 7f7cb3a68540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-06-23T01:30:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:57.285+0000 7f7cb3a68540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-06-23T01:30:57.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:57.285+0000 7f7cb3a68540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:30:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:57.285+0000 7f7cb3a68540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:30:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:57.285+0000 7f7cb3a68540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-06-23T01:30:57.289 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:57.286+0000 7f7cb3a68540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-06-23T01:30:57.811 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:57.809+0000 7f7cb3a68540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:30:58.576 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:58.574+0000 7f7cb3a68540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:30:59.336 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:30:59.335+0000 7f7cb3a68540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:00.130 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-06-23T01:31:00.156 DEBUG:teuthology.orchestra.run.smithi092:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:31:00.241 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:00.239+0000 7f41e672d540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-06-23T01:31:00.241 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:00.239+0000 7f41e672d540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-06-23T01:31:00.241 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:00.239+0000 7f41e672d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:31:00.241 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:00.239+0000 7f41e672d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:31:00.241 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:00.239+0000 7f41e672d540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-06-23T01:31:00.241 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:00.240+0000 7f41e672d540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-06-23T01:31:00.762 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:00.760+0000 7f41e672d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:01.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:01.521+0000 7f41e672d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:02.288 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:02.287+0000 7f41e672d540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:03.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-06-23T01:31:03.149 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-06-23T01:31:03.176 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:31:03.177 INFO:tasks.ceph:role: osd.4 2023-06-23T01:31:03.177 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:03.177 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-06-23T01:31:03.251 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:31:03.251 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:31:03.251 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:31:03.251 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 2023-06-23T01:31:03.252 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:31:03.252 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:31:03.252 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:31:03.252 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:31:03.252 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:31:03.252 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:31:03.684 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2023-06-23T01:31:03.686 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2023-06-23T01:31:03.687 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-06-23T01:31:03.859 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-06-23T01:31:03.891 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-06-23T01:31:03.957 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:31:03.958 INFO:tasks.ceph:role: osd.5 2023-06-23T01:31:03.958 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:03.959 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-06-23T01:31:04.030 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:31:04.030 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:31:04.030 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:31:04.030 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 2023-06-23T01:31:04.030 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:31:04.030 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:31:04.031 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:31:04.031 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:31:04.031 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:31:04.031 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:31:04.458 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2023-06-23T01:31:04.459 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2023-06-23T01:31:04.460 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-06-23T01:31:04.562 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-06-23T01:31:04.590 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-06-23T01:31:04.654 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:31:04.655 INFO:tasks.ceph:role: osd.6 2023-06-23T01:31:04.655 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:04.656 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-06-23T01:31:04.726 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:31:04.727 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:31:04.727 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:31:04.727 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 2023-06-23T01:31:04.727 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:31:04.727 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:31:04.728 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:31:04.728 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:31:04.728 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:31:04.728 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:31:05.174 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2023-06-23T01:31:05.175 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2023-06-23T01:31:05.176 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-06-23T01:31:05.345 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-06-23T01:31:05.375 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-06-23T01:31:05.439 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-06-23T01:31:05.440 INFO:tasks.ceph:role: osd.7 2023-06-23T01:31:05.440 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:05.441 DEBUG:teuthology.orchestra.run.smithi119:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-06-23T01:31:05.511 INFO:teuthology.orchestra.run.smithi119.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-06-23T01:31:05.512 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 attr=2, projid32bit=1 2023-06-23T01:31:05.512 INFO:teuthology.orchestra.run.smithi119.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-06-23T01:31:05.512 INFO:teuthology.orchestra.run.smithi119.stdout: = reflink=1 2023-06-23T01:31:05.513 INFO:teuthology.orchestra.run.smithi119.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-06-23T01:31:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: = sunit=0 swidth=0 blks 2023-06-23T01:31:05.513 INFO:teuthology.orchestra.run.smithi119.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-06-23T01:31:05.513 INFO:teuthology.orchestra.run.smithi119.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-06-23T01:31:05.513 INFO:teuthology.orchestra.run.smithi119.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-06-23T01:31:05.513 INFO:teuthology.orchestra.run.smithi119.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-06-23T01:31:05.970 INFO:teuthology.orchestra.run.smithi119.stdout:Discarding blocks...Done. 2023-06-23T01:31:05.972 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi119.front.sepia.ceph.com -o noatime 2023-06-23T01:31:05.972 DEBUG:teuthology.orchestra.run.smithi119:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-06-23T01:31:06.137 DEBUG:teuthology.orchestra.run.smithi119:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-06-23T01:31:06.169 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:31:06.255 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:06.252+0000 7f8776e30540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-06-23T01:31:06.256 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:06.252+0000 7f8776e30540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-06-23T01:31:06.256 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:06.252+0000 7f8776e30540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:31:06.256 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:06.252+0000 7f8776e30540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:31:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:06.252+0000 7f8776e30540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-06-23T01:31:06.257 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:06.253+0000 7f8776e30540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-06-23T01:31:06.780 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:06.777+0000 7f8776e30540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:07.541 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:07.539+0000 7f8776e30540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:08.304 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:08.302+0000 7f8776e30540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:09.103 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-06-23T01:31:09.132 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:31:09.217 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:09.215+0000 7fc377873540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-06-23T01:31:09.218 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:09.215+0000 7fc377873540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-06-23T01:31:09.218 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:09.215+0000 7fc377873540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:31:09.218 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:09.215+0000 7fc377873540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:31:09.218 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:09.215+0000 7fc377873540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-06-23T01:31:09.219 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:09.216+0000 7fc377873540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-06-23T01:31:09.739 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:09.737+0000 7fc377873540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:10.500 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:10.497+0000 7fc377873540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:11.262 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:11.260+0000 7fc377873540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:12.053 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-06-23T01:31:12.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:31:12.165 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:12.162+0000 7efd30391540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-06-23T01:31:12.166 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:12.162+0000 7efd30391540 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-06-23T01:31:12.166 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:12.162+0000 7efd30391540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:31:12.166 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:12.162+0000 7efd30391540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:31:12.167 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:12.162+0000 7efd30391540 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-06-23T01:31:12.167 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:12.163+0000 7efd30391540 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-06-23T01:31:12.690 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:12.687+0000 7efd30391540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:13.455 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:13.454+0000 7efd30391540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:14.216 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:14.215+0000 7efd30391540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:15.012 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-06-23T01:31:15.081 DEBUG:teuthology.orchestra.run.smithi119:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:31:15.082 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:31:15.088 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:31:15.164 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:15.162+0000 7f51081f6540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-06-23T01:31:15.165 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:15.163+0000 7f51081f6540 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-06-23T01:31:15.165 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:15.163+0000 7f51081f6540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:31:15.165 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:15.163+0000 7f51081f6540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:31:15.165 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:15.163+0000 7f51081f6540 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-06-23T01:31:15.165 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:15.163+0000 7f51081f6540 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-06-23T01:31:15.686 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:15.685+0000 7f51081f6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:16.447 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:16.446+0000 7f51081f6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:17.210 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:17.208+0000 7f51081f6540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-06-23T01:31:18.005 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-06-23T01:31:18.034 INFO:tasks.ceph:Reading keys from all nodes... 2023-06-23T01:31:18.034 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.035 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-06-23T01:31:18.061 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.061 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-06-23T01:31:18.128 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.129 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-06-23T01:31:18.194 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.195 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-06-23T01:31:18.259 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.259 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-06-23T01:31:18.324 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.324 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-06-23T01:31:18.388 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.389 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-06-23T01:31:18.457 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.457 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-06-23T01:31:18.486 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.487 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-06-23T01:31:18.557 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.558 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-06-23T01:31:18.626 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.626 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-06-23T01:31:18.692 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.693 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-06-23T01:31:18.758 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.759 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-06-23T01:31:18.824 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.824 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-06-23T01:31:18.889 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.889 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-06-23T01:31:18.913 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:18.914 DEBUG:teuthology.orchestra.run.smithi092:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-06-23T01:31:18.968 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.969 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-06-23T01:31:18.985 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:18.985 DEBUG:teuthology.orchestra.run.smithi119:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-06-23T01:31:19.040 INFO:tasks.ceph:Adding keys to all mons... 2023-06-23T01:31:19.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T01:31:19.044 DEBUG:teuthology.orchestra.run.smithi119:> sudo tee -a /etc/ceph/ceph.keyring 2023-06-23T01:31:19.083 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.y] 2023-06-23T01:31:19.083 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDH9ZRkZR7sHhAA/G5QPQVzDQpD08zZXLLKmA== 2023-06-23T01:31:19.083 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.a] 2023-06-23T01:31:19.083 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDH9ZRk8giLJhAAmdVpgscGQ3uxPVg8f+cpSg== 2023-06-23T01:31:19.084 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.c] 2023-06-23T01:31:19.084 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDH9ZRkQclHLRAAjipOb586ObMOvwXNWxY8WQ== 2023-06-23T01:31:19.084 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.0] 2023-06-23T01:31:19.084 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDL9ZRkbkuWFRAAscYQ3DDruxSY3uM7bKo09A== 2023-06-23T01:31:19.084 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.1] 2023-06-23T01:31:19.084 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDO9ZRkEQ0tExAAzz4uJB1e/SOouzkz2dr3Aw== 2023-06-23T01:31:19.085 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.2] 2023-06-23T01:31:19.085 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDR9ZRk5sgcERAAuqYus4fT9lDpbAj4Fu2ymA== 2023-06-23T01:31:19.085 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.3] 2023-06-23T01:31:19.085 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDU9ZRkELdLDhAAAQlVe5Tqb95iYRoRZdsFCA== 2023-06-23T01:31:19.086 INFO:teuthology.orchestra.run.smithi092.stdout:[mgr.x] 2023-06-23T01:31:19.086 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDH9ZRkfGiFIxAAp2cpYxaH4Xspw0XzyOo5pA== 2023-06-23T01:31:19.086 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.b] 2023-06-23T01:31:19.087 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDH9ZRkY4WmMRAAE2FRiZ4PWl18F0Hf3z/5KQ== 2023-06-23T01:31:19.087 INFO:teuthology.orchestra.run.smithi092.stdout:[mds.d] 2023-06-23T01:31:19.087 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDH9ZRklBddOBAArmT8oKVC403kRyyMkvgodg== 2023-06-23T01:31:19.087 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.4] 2023-06-23T01:31:19.088 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDa9ZRk3MokDxAAVFGX4KtHCv2NNt8V7zQPzw== 2023-06-23T01:31:19.088 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.5] 2023-06-23T01:31:19.088 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDd9ZRkhlntDBAAjzkxN0gmDIS9BoeGKhdAtQ== 2023-06-23T01:31:19.088 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.6] 2023-06-23T01:31:19.089 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDg9ZRkLGLGCRAAi0tsNYKIKmLjrTsL6Yv+zw== 2023-06-23T01:31:19.089 INFO:teuthology.orchestra.run.smithi092.stdout:[osd.7] 2023-06-23T01:31:19.089 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDj9ZRke0C8CRAAv5KgwR/8KZz9H4Q1HuCf/Q== 2023-06-23T01:31:19.090 INFO:teuthology.orchestra.run.smithi092.stdout:[client.0] 2023-06-23T01:31:19.090 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDI9ZRkXXWmABAA4xFIhIfg6zgFSFoGr6IKFg== 2023-06-23T01:31:19.090 INFO:teuthology.orchestra.run.smithi092.stdout:[client.2] 2023-06-23T01:31:19.091 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDI9ZRk0Yb2BRAAaNGxPETE5uWB0g3AmdKKYg== 2023-06-23T01:31:19.091 INFO:teuthology.orchestra.run.smithi092.stdout:[client.1] 2023-06-23T01:31:19.091 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDI9ZRkbSr1CBAAONAOF3k3jDGinBnsQD9QPg== 2023-06-23T01:31:19.091 INFO:teuthology.orchestra.run.smithi092.stdout:[client.3] 2023-06-23T01:31:19.092 INFO:teuthology.orchestra.run.smithi092.stdout: key = AQDI9ZRkzqlkDhAARpebCkQZecvgCYreZIzOpg== 2023-06-23T01:31:19.103 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.y] 2023-06-23T01:31:19.104 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDH9ZRkZR7sHhAA/G5QPQVzDQpD08zZXLLKmA== 2023-06-23T01:31:19.104 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.a] 2023-06-23T01:31:19.104 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDH9ZRk8giLJhAAmdVpgscGQ3uxPVg8f+cpSg== 2023-06-23T01:31:19.104 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.c] 2023-06-23T01:31:19.104 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDH9ZRkQclHLRAAjipOb586ObMOvwXNWxY8WQ== 2023-06-23T01:31:19.105 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.0] 2023-06-23T01:31:19.105 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDL9ZRkbkuWFRAAscYQ3DDruxSY3uM7bKo09A== 2023-06-23T01:31:19.105 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.1] 2023-06-23T01:31:19.105 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDO9ZRkEQ0tExAAzz4uJB1e/SOouzkz2dr3Aw== 2023-06-23T01:31:19.105 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.2] 2023-06-23T01:31:19.105 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDR9ZRk5sgcERAAuqYus4fT9lDpbAj4Fu2ymA== 2023-06-23T01:31:19.106 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.3] 2023-06-23T01:31:19.106 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDU9ZRkELdLDhAAAQlVe5Tqb95iYRoRZdsFCA== 2023-06-23T01:31:19.106 INFO:teuthology.orchestra.run.smithi119.stdout:[mgr.x] 2023-06-23T01:31:19.106 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDH9ZRkfGiFIxAAp2cpYxaH4Xspw0XzyOo5pA== 2023-06-23T01:31:19.106 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.b] 2023-06-23T01:31:19.106 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDH9ZRkY4WmMRAAE2FRiZ4PWl18F0Hf3z/5KQ== 2023-06-23T01:31:19.106 INFO:teuthology.orchestra.run.smithi119.stdout:[mds.d] 2023-06-23T01:31:19.107 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDH9ZRklBddOBAArmT8oKVC403kRyyMkvgodg== 2023-06-23T01:31:19.107 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.4] 2023-06-23T01:31:19.107 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDa9ZRk3MokDxAAVFGX4KtHCv2NNt8V7zQPzw== 2023-06-23T01:31:19.107 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.5] 2023-06-23T01:31:19.107 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDd9ZRkhlntDBAAjzkxN0gmDIS9BoeGKhdAtQ== 2023-06-23T01:31:19.107 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.6] 2023-06-23T01:31:19.107 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDg9ZRkLGLGCRAAi0tsNYKIKmLjrTsL6Yv+zw== 2023-06-23T01:31:19.107 INFO:teuthology.orchestra.run.smithi119.stdout:[osd.7] 2023-06-23T01:31:19.108 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDj9ZRke0C8CRAAv5KgwR/8KZz9H4Q1HuCf/Q== 2023-06-23T01:31:19.108 INFO:teuthology.orchestra.run.smithi119.stdout:[client.0] 2023-06-23T01:31:19.108 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDI9ZRkXXWmABAA4xFIhIfg6zgFSFoGr6IKFg== 2023-06-23T01:31:19.108 INFO:teuthology.orchestra.run.smithi119.stdout:[client.2] 2023-06-23T01:31:19.108 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDI9ZRk0Yb2BRAAaNGxPETE5uWB0g3AmdKKYg== 2023-06-23T01:31:19.108 INFO:teuthology.orchestra.run.smithi119.stdout:[client.1] 2023-06-23T01:31:19.108 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDI9ZRkbSr1CBAAONAOF3k3jDGinBnsQD9QPg== 2023-06-23T01:31:19.109 INFO:teuthology.orchestra.run.smithi119.stdout:[client.3] 2023-06-23T01:31:19.109 INFO:teuthology.orchestra.run.smithi119.stdout: key = AQDI9ZRkzqlkDhAARpebCkQZecvgCYreZIzOpg== 2023-06-23T01:31:19.110 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:31:19.135 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:31:19.197 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:31:19.199 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:31:19.249 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:31:19.251 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:31:19.301 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.304 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.352 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.354 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.405 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.453 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.455 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.501 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:31:19.503 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-06-23T01:31:19.550 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:31:19.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:31:19.602 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:31:19.604 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-06-23T01:31:19.653 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.656 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.703 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.705 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.751 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.754 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.800 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.803 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-06-23T01:31:19.849 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:31:19.852 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:31:19.902 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:31:19.904 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:31:19.954 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:31:19.956 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:31:20.005 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:31:20.008 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-06-23T01:31:20.058 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-06-23T01:31:20.058 DEBUG:teuthology.orchestra.run.smithi092:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-06-23T01:31:20.083 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:31:20.174 DEBUG:teuthology.orchestra.run.smithi092:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-06-23T01:31:20.198 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-06-23T01:31:20.222 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:31:20.313 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-06-23T01:31:20.337 DEBUG:teuthology.orchestra.run.smithi119:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-06-23T01:31:20.401 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-06-23T01:31:20.495 DEBUG:teuthology.orchestra.run.smithi119:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-06-23T01:31:20.518 DEBUG:teuthology.orchestra.run.smithi092:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:31:20.520 DEBUG:teuthology.orchestra.run.smithi119:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-06-23T01:31:20.572 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-06-23T01:31:20.573 INFO:tasks.ceph.mon.a:Restarting daemon 2023-06-23T01:31:20.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-06-23T01:31:20.575 INFO:tasks.ceph.mon.a:Started 2023-06-23T01:31:20.575 INFO:tasks.ceph.mon.b:Restarting daemon 2023-06-23T01:31:20.576 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-06-23T01:31:20.614 INFO:tasks.ceph.mon.b:Started 2023-06-23T01:31:20.615 INFO:tasks.ceph.mon.c:Restarting daemon 2023-06-23T01:31:20.615 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-06-23T01:31:20.617 INFO:tasks.ceph.mon.c:Started 2023-06-23T01:31:20.617 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-06-23T01:31:20.618 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-06-23T01:31:20.618 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-06-23T01:31:20.620 INFO:tasks.ceph.mgr.y:Started 2023-06-23T01:31:20.620 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-06-23T01:31:20.621 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-06-23T01:31:20.623 INFO:tasks.ceph.mgr.x:Started 2023-06-23T01:31:20.623 INFO:tasks.ceph:Setting crush tunables to default 2023-06-23T01:31:20.624 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd crush tunables default 2023-06-23T01:31:20.740 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.737+0000 7ff4d31b3700 1 Processor -- start 2023-06-23T01:31:20.740 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.737+0000 7ff4d31b3700 1 -- start start 2023-06-23T01:31:20.741 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4d31b3700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7ff4cc1570b0 0x7ff4cc15a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:20.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4d31b3700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 0x7ff4cc15d300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:20.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4d31b3700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4cc0d03a0 con 0x7ff4cc0b2010 2023-06-23T01:31:20.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4d31b3700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4cc0cef00 con 0x7ff4cc157480 2023-06-23T01:31:20.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4d31b3700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4cc0cec00 con 0x7ff4cc1570b0 2023-06-23T01:31:20.742 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4c3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 0x7ff4cc15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:20.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4c3fff700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 0x7ff4cc15d300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:59562/0 (socket says 172.21.15.92:59562) 2023-06-23T01:31:20.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4c3fff700 1 -- 172.21.15.92:0/4089171694 learned_addr learned my addr 172.21.15.92:0/4089171694 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:20.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4c3fff700 1 -- 172.21.15.92:0/4089171694 >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 msgr2=0x7ff4cc15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-06-23T01:31:20.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4c3fff700 1 -- 172.21.15.92:0/4089171694 >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 msgr2=0x7ff4cc15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:31:20.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4c3fff700 1 --2- 172.21.15.92:0/4089171694 >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 0x7ff4cc15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:31:20.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4c3fff700 1 --2- 172.21.15.92:0/4089171694 >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 0x7ff4cc15d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:31:20.743 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.738+0000 7ff4d0f4f700 1 --2- 172.21.15.92:0/4089171694 >> v2:172.21.15.119:3300/0 conn(0x7ff4cc1570b0 0x7ff4cc15a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:20.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.739+0000 7ff4d0f4f700 1 -- 172.21.15.92:0/4089171694 >> v2:172.21.15.119:3300/0 conn(0x7ff4cc1570b0 msgr2=0x7ff4cc15a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-06-23T01:31:20.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.739+0000 7ff4d0f4f700 1 -- 172.21.15.92:0/4089171694 >> v2:172.21.15.119:3300/0 conn(0x7ff4cc1570b0 msgr2=0x7ff4cc15a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-06-23T01:31:20.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.739+0000 7ff4d0f4f700 1 --2- 172.21.15.92:0/4089171694 >> v2:172.21.15.119:3300/0 conn(0x7ff4cc1570b0 0x7ff4cc15a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-06-23T01:31:20.744 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.739+0000 7ff4d0f4f700 1 --2- 172.21.15.92:0/4089171694 >> v2:172.21.15.119:3300/0 conn(0x7ff4cc1570b0 0x7ff4cc15a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-06-23T01:31:20.935 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.931+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764523117 0 0) 0x7ff4cc0d03a0 con 0x7ff4cc0b2010 2023-06-23T01:31:20.936 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.931+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4b4003570 con 0x7ff4cc0b2010 2023-06-23T01:31:20.936 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 <== mon.1 v1:172.21.15.119:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3114388839 0 0) 0x7ff4b4003570 con 0x7ff4cc0b2010 2023-06-23T01:31:20.936 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 >> v2:172.21.15.119:3300/0 conn(0x7ff4cc1570b0 msgr2=0x7ff4cc15a8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-23T01:31:20.936 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 --2- 172.21.15.92:0/4089171694 >> v2:172.21.15.119:3300/0 conn(0x7ff4cc1570b0 0x7ff4cc15a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:20.936 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 msgr2=0x7ff4cc15d300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-06-23T01:31:20.936 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 --2- 172.21.15.92:0/4089171694 >> v2:172.21.15.92:3300/0 conn(0x7ff4cc157480 0x7ff4cc15d300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:20.937 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 --> v1:172.21.15.119:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc15db50 con 0x7ff4cc0b2010 2023-06-23T01:31:20.937 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 <== mon.1 v1:172.21.15.119:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3236472324 0 0) 0x7ff4c4001470 con 0x7ff4cc0b2010 2023-06-23T01:31:20.937 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 >> v1:172.21.15.119:6789/0 conn(0x7ff4cc0b2010 legacy=0x7ff4cc0d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:20.937 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4b4003a60 0x7ff4b4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:20.937 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4b4004810 0x7ff4b4004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:20.937 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b40051c0 0x7ff4b400ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:20.937 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0cef00 con 0x7ff4b4003a60 2023-06-23T01:31:20.938 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc0cec00 con 0x7ff4b40051c0 2023-06-23T01:31:20.938 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4b400c130 con 0x7ff4b4004810 2023-06-23T01:31:20.938 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d0f4f700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4b4003a60 0x7ff4b4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:20.938 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d1750700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b40051c0 0x7ff4b400ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:20.938 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c3fff700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4b4004810 0x7ff4b4004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:20.938 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d0f4f700 1 -- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4b4004810 msgr2=0x7ff4b4004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:20.938 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d0f4f700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4b4004810 0x7ff4b4004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:20.938 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d0f4f700 1 -- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b40051c0 msgr2=0x7ff4b400ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:20.939 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d0f4f700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b40051c0 0x7ff4b400ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:20.939 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d0f4f700 1 -- 172.21.15.92:0/4089171694 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4b4003570 con 0x7ff4b4003a60 2023-06-23T01:31:20.939 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d1750700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b40051c0 0x7ff4b400ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:31:20.939 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4c3fff700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4b4004810 0x7ff4b4004c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:31:20.939 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d1750700 1 -- 172.21.15.92:0/4089171694 reap_dead start 2023-06-23T01:31:20.939 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.932+0000 7ff4d0f4f700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4b4003a60 0x7ff4b4003e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8007440 tx=0x7ff4c8009720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=211f6df7285bd3ac server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:20.940 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.933+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8009a30 con 0x7ff4b4003a60 2023-06-23T01:31:20.940 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.933+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4c8007cf0 con 0x7ff4b4003a60 2023-06-23T01:31:20.940 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.933+0000 7ff4c37fe700 1 -- 172.21.15.92:0/4089171694 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4c8014630 con 0x7ff4b4003a60 2023-06-23T01:31:20.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.934+0000 7ff4d31b3700 1 -- 172.21.15.92:0/4089171694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4b4003a60 msgr2=0x7ff4b4003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:20.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.934+0000 7ff4d31b3700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4b4003a60 0x7ff4b4003e70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8007440 tx=0x7ff4c8009720 comp rx=0 tx=0).stop 2023-06-23T01:31:20.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.934+0000 7ff4d31b3700 1 -- 172.21.15.92:0/4089171694 shutdown_connections 2023-06-23T01:31:20.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.934+0000 7ff4d31b3700 1 --2- 172.21.15.92:0/4089171694 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4b4003a60 0x7ff4b4003e70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:20.942 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.934+0000 7ff4d31b3700 1 -- 172.21.15.92:0/4089171694 >> 172.21.15.92:0/4089171694 conn(0x7ff4cc070290 msgr2=0x7ff4cc070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:20.943 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.934+0000 7ff4d31b3700 1 -- 172.21.15.92:0/4089171694 shutdown_connections 2023-06-23T01:31:20.943 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.934+0000 7ff4d31b3700 1 -- 172.21.15.92:0/4089171694 wait complete. 2023-06-23T01:31:20.943 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.935+0000 7ff4d31b3700 1 Processor -- start 2023-06-23T01:31:20.943 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.935+0000 7ff4d31b3700 1 -- start start 2023-06-23T01:31:20.943 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.936+0000 7ff4d31b3700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b4003a60 0x7ff4cc159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:20.943 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.936+0000 7ff4d31b3700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4cc14bb70 0x7ff4cc14d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:20.944 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.936+0000 7ff4d31b3700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4cc14a530 0x7ff4cc153460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:20.944 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.936+0000 7ff4d31b3700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc159b40 con 0x7ff4cc14bb70 2023-06-23T01:31:20.944 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.936+0000 7ff4d31b3700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc15bce0 con 0x7ff4b4003a60 2023-06-23T01:31:20.944 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.936+0000 7ff4d31b3700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4cc15ae30 con 0x7ff4cc14a530 2023-06-23T01:31:20.944 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4cc14bb70 0x7ff4cc14d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:20.944 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4cc14bb70 0x7ff4cc14d2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:59596/0 (socket says 172.21.15.92:59596) 2023-06-23T01:31:20.945 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4d0f4f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b4003a60 0x7ff4cc159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:20.945 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 -- 172.21.15.92:0/94534984 learned_addr learned my addr 172.21.15.92:0/94534984 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:20.945 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4d1750700 1 --2- 172.21.15.92:0/94534984 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4cc14a530 0x7ff4cc153460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:20.945 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 -- 172.21.15.92:0/94534984 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4cc14a530 msgr2=0x7ff4cc153460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:20.945 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 --2- 172.21.15.92:0/94534984 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4cc14a530 0x7ff4cc153460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:20.945 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 -- 172.21.15.92:0/94534984 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b4003a60 msgr2=0x7ff4cc159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:20.946 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 --2- 172.21.15.92:0/94534984 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b4003a60 0x7ff4cc159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:20.946 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 -- 172.21.15.92:0/94534984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4cc15b0d0 con 0x7ff4cc14bb70 2023-06-23T01:31:20.946 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.937+0000 7ff4c3fff700 1 --2- 172.21.15.92:0/94534984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4cc14bb70 0x7ff4cc14d2b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0109d0 tx=0x7ff4bc008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:20.946 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.938+0000 7ff4c0ff9700 1 -- 172.21.15.92:0/94534984 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc010ec0 con 0x7ff4cc14bb70 2023-06-23T01:31:20.946 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.938+0000 7ff4c0ff9700 1 -- 172.21.15.92:0/94534984 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4bc00c050 con 0x7ff4cc14bb70 2023-06-23T01:31:20.946 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.939+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4cc15b3b0 con 0x7ff4cc14bb70 2023-06-23T01:31:20.946 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.939+0000 7ff4c0ff9700 1 -- 172.21.15.92:0/94534984 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4bc00ea20 con 0x7ff4cc14bb70 2023-06-23T01:31:20.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.939+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4cc15b970 con 0x7ff4cc14bb70 2023-06-23T01:31:20.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.939+0000 7ff4c0ff9700 1 -- 172.21.15.92:0/94534984 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff4bc015470 con 0x7ff4cc14bb70 2023-06-23T01:31:20.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.939+0000 7ff4c0ff9700 1 -- 172.21.15.92:0/94534984 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff4bc016770 con 0x7ff4cc14bb70 2023-06-23T01:31:20.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.940+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4cc0af3e0 con 0x7ff4cc14bb70 2023-06-23T01:31:20.947 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:20.942+0000 7ff4c0ff9700 1 -- 172.21.15.92:0/94534984 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7ff4bc01a030 con 0x7ff4cc14bb70 2023-06-23T01:31:21.020 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.017+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff4cc04c1b0 con 0x7ff4cc14bb70 2023-06-23T01:31:21.027 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.024+0000 7ff4c0ff9700 1 -- 172.21.15.92:0/94534984 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff4bc02c090 con 0x7ff4cc14bb70 2023-06-23T01:31:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.025+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4cc14bb70 msgr2=0x7ff4cc14d2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.028 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.025+0000 7ff4d31b3700 1 --2- 172.21.15.92:0/94534984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4cc14bb70 0x7ff4cc14d2b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff4bc0109d0 tx=0x7ff4bc008040 comp rx=0 tx=0).stop 2023-06-23T01:31:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.025+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 shutdown_connections 2023-06-23T01:31:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.025+0000 7ff4d31b3700 1 --2- 172.21.15.92:0/94534984 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7ff4cc14a530 0x7ff4cc153460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.025+0000 7ff4d31b3700 1 --2- 172.21.15.92:0/94534984 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7ff4b4003a60 0x7ff4cc159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.025+0000 7ff4d31b3700 1 --2- 172.21.15.92:0/94534984 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7ff4cc14bb70 0x7ff4cc14d2b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.025+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 >> 172.21.15.92:0/94534984 conn(0x7ff4cc070290 msgr2=0x7ff4cc0b4800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.026+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 shutdown_connections 2023-06-23T01:31:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:21.026+0000 7ff4d31b3700 1 -- 172.21.15.92:0/94534984 wait complete. 2023-06-23T01:31:21.029 INFO:teuthology.orchestra.run.smithi092.stderr:adjusted tunables profile to default 2023-06-23T01:31:21.040 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-06-23T01:31:21.041 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:21.041 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-06-23T01:31:21.065 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:21.065 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-06-23T01:31:21.129 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:21.130 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-06-23T01:31:21.193 DEBUG:teuthology.orchestra.run.smithi092:> set -ex 2023-06-23T01:31:21.194 DEBUG:teuthology.orchestra.run.smithi092:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-06-23T01:31:21.257 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:21.257 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-06-23T01:31:21.281 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:21.282 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-06-23T01:31:21.346 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:21.346 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-06-23T01:31:21.410 DEBUG:teuthology.orchestra.run.smithi119:> set -ex 2023-06-23T01:31:21.410 DEBUG:teuthology.orchestra.run.smithi119:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-06-23T01:31:21.474 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 4bcf4cdc-7c98-4a4a-bc56-133f22f4d802 0 2023-06-23T01:31:21.628 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.627+0000 7f0585a07700 1 Processor -- start 2023-06-23T01:31:21.629 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.627+0000 7f0585a07700 1 -- start start 2023-06-23T01:31:21.630 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f0585a07700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f058006ba50 0x7f058015a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.631 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f0585a07700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006be20 0x7f05800c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.631 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f0585a07700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f05800c74b0 0x7f0580068420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.631 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f0585a07700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058015a540 con 0x7f058006ba50 2023-06-23T01:31:21.631 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f0585a07700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05801595b0 con 0x7f05800c74b0 2023-06-23T01:31:21.631 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f0585a07700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0580159b70 con 0x7f058006be20 2023-06-23T01:31:21.631 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006be20 0x7f05800c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.631 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006be20 0x7f05800c6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:36994/0 (socket says 172.21.15.119:36994) 2023-06-23T01:31:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 -- 172.21.15.119:0/168175773 learned_addr learned my addr 172.21.15.119:0/168175773 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057f7fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f05800c74b0 0x7f0580068420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057effd700 1 --2- 172.21.15.119:0/168175773 >> v2:172.21.15.92:3300/0 conn(0x7f058006ba50 0x7f058015a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 -- 172.21.15.119:0/168175773 >> v2:172.21.15.119:3300/0 conn(0x7f05800c74b0 msgr2=0x7f0580068420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 --2- 172.21.15.119:0/168175773 >> v2:172.21.15.119:3300/0 conn(0x7f05800c74b0 0x7f0580068420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 -- 172.21.15.119:0/168175773 >> v2:172.21.15.92:3300/0 conn(0x7f058006ba50 msgr2=0x7f058015a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:21.632 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 --2- 172.21.15.119:0/168175773 >> v2:172.21.15.92:3300/0 conn(0x7f058006ba50 0x7f058015a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 -- 172.21.15.119:0/168175773 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05800c79c0 con 0x7f058006be20 2023-06-23T01:31:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057effd700 1 --2- 172.21.15.119:0/168175773 >> v2:172.21.15.92:3300/0 conn(0x7f058006ba50 0x7f058015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:31:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.628+0000 7f057e7fc700 1 --2- 172.21.15.119:0/168175773 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006be20 0x7f05800c6f70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f05800cc8d0 tx=0x7f0570009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f00401716deeebf server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f057dffb700 1 -- 172.21.15.119:0/168175773 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0570009e70 con 0x7f058006be20 2023-06-23T01:31:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f057dffb700 1 -- 172.21.15.119:0/168175773 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057000b860 con 0x7f058006be20 2023-06-23T01:31:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f057dffb700 1 -- 172.21.15.119:0/168175773 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f057000bc00 con 0x7f058006be20 2023-06-23T01:31:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 -- 172.21.15.119:0/168175773 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006be20 msgr2=0x7f05800c6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 --2- 172.21.15.119:0/168175773 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006be20 0x7f05800c6f70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f05800cc8d0 tx=0x7f0570009250 comp rx=0 tx=0).stop 2023-06-23T01:31:21.634 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 -- 172.21.15.119:0/168175773 shutdown_connections 2023-06-23T01:31:21.634 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 --2- 172.21.15.119:0/168175773 >> v2:172.21.15.119:3300/0 conn(0x7f05800c74b0 0x7f0580068420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.634 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 --2- 172.21.15.119:0/168175773 >> v2:172.21.15.92:3300/0 conn(0x7f058006ba50 0x7f058015a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.634 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 --2- 172.21.15.119:0/168175773 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006be20 0x7f05800c6f70 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.634 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 -- 172.21.15.119:0/168175773 >> 172.21.15.119:0/168175773 conn(0x7f05800c9ea0 msgr2=0x7f058006d730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:21.634 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 -- 172.21.15.119:0/168175773 shutdown_connections 2023-06-23T01:31:21.634 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.629+0000 7f0585a07700 1 -- 172.21.15.119:0/168175773 wait complete. 2023-06-23T01:31:21.636 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.630+0000 7f0585a07700 1 Processor -- start 2023-06-23T01:31:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f0585a07700 1 -- start start 2023-06-23T01:31:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f0585a07700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006ba50 0x7f0580154550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f0585a07700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f058006be20 0x7f05800ac6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f0585a07700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05800c74b0 0x7f05800acc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f0585a07700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05801677a0 con 0x7f058006be20 2023-06-23T01:31:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f0585a07700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0580154a90 con 0x7f05800c74b0 2023-06-23T01:31:21.637 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f0585a07700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0580166e70 con 0x7f058006ba50 2023-06-23T01:31:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f057effd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006ba50 0x7f0580154550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f057effd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006ba50 0x7f0580154550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:36996/0 (socket says 172.21.15.119:36996) 2023-06-23T01:31:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f057effd700 1 -- 172.21.15.119:0/1308744909 learned_addr learned my addr 172.21.15.119:0/1308744909 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f057f7fe700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05800c74b0 0x7f05800acc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.631+0000 7f057e7fc700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f058006be20 0x7f05800ac6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f057effd700 1 -- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05800c74b0 msgr2=0x7f05800acc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.638 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f057effd700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05800c74b0 0x7f05800acc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f057effd700 1 -- 172.21.15.119:0/1308744909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f058006be20 msgr2=0x7f05800ac6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f057effd700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f058006be20 0x7f05800ac6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f057effd700 1 -- 172.21.15.119:0/1308744909 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05700094f0 con 0x7f058006ba50 2023-06-23T01:31:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f057f7fe700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05800c74b0 0x7f05800acc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f057effd700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006ba50 0x7f0580154550 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f057400fc10 tx=0x7f0574008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f0566ffd700 1 -- 172.21.15.119:0/1308744909 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05740078f0 con 0x7f058006ba50 2023-06-23T01:31:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f0566ffd700 1 -- 172.21.15.119:0/1308744909 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0574007a50 con 0x7f058006ba50 2023-06-23T01:31:21.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0580167150 con 0x7f058006ba50 2023-06-23T01:31:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f0566ffd700 1 -- 172.21.15.119:0/1308744909 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574007df0 con 0x7f058006ba50 2023-06-23T01:31:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0580166b80 con 0x7f058006ba50 2023-06-23T01:31:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f0566ffd700 1 -- 172.21.15.119:0/1308744909 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f05740118e0 con 0x7f058006ba50 2023-06-23T01:31:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.632+0000 7f0566ffd700 1 -- 172.21.15.119:0/1308744909 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7f057401ba40 con 0x7f058006ba50 2023-06-23T01:31:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.633+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058006c690 con 0x7f058006ba50 2023-06-23T01:31:21.640 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.635+0000 7f0566ffd700 1 -- 172.21.15.119:0/1308744909 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f0574020060 con 0x7f058006ba50 2023-06-23T01:31:21.719 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.718+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4bcf4cdc-7c98-4a4a-bc56-133f22f4d802", "id": 0} v 0) v1 -- 0x7f05800c7ca0 con 0x7f058006ba50 2023-06-23T01:31:21.724 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.723+0000 7f0566ffd700 1 -- 172.21.15.119:0/1308744909 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4bcf4cdc-7c98-4a4a-bc56-133f22f4d802", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0574011cd0 con 0x7f058006ba50 2023-06-23T01:31:21.725 INFO:teuthology.orchestra.run.smithi119.stdout:0 2023-06-23T01:31:21.725 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006ba50 msgr2=0x7f0580154550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006ba50 0x7f0580154550 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f057400fc10 tx=0x7f0574008040 comp rx=0 tx=0).stop 2023-06-23T01:31:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 shutdown_connections 2023-06-23T01:31:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f05800c74b0 0x7f05800acc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f058006ba50 0x7f0580154550 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 --2- 172.21.15.119:0/1308744909 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f058006be20 0x7f05800ac6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 >> 172.21.15.119:0/1308744909 conn(0x7f05800c9ea0 msgr2=0x7f05800c8010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 shutdown_connections 2023-06-23T01:31:21.726 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.724+0000 7f0585a07700 1 -- 172.21.15.119:0/1308744909 wait complete. 2023-06-23T01:31:21.737 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new a2e05a19-0c70-4cb4-aacd-3381219435b8 1 2023-06-23T01:31:21.852 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.851+0000 7fc8e7115700 1 Processor -- start 2023-06-23T01:31:21.852 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.851+0000 7fc8e7115700 1 -- start start 2023-06-23T01:31:21.853 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.851+0000 7fc8e7115700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153e50 0x7fc8e00d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.854 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.851+0000 7fc8e7115700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc8e00b2010 0x7fc8e00b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.854 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.851+0000 7fc8e7115700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8e00cedd0 con 0x7fc8e0153a80 2023-06-23T01:31:21.854 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.851+0000 7fc8e7115700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8e00d14c0 con 0x7fc8e00b2010 2023-06-23T01:31:21.854 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8e7115700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e00b29d0 con 0x7fc8e0153e50 2023-06-23T01:31:21.854 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8e56b2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc8e00b2010 0x7fc8e00b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.854 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8dffff700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153e50 0x7fc8e00d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.854 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8e56b2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc8e00b2010 0x7fc8e00b2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51408/0 (socket says 172.21.15.119:51408) 2023-06-23T01:31:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8e56b2700 1 -- 172.21.15.119:0/3907001019 learned_addr learned my addr 172.21.15.119:0/3907001019 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8dffff700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153e50 0x7fc8e00d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:37012/0 (socket says 172.21.15.119:37012) 2023-06-23T01:31:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8dffff700 1 -- 172.21.15.119:0/3907001019 >> v2:172.21.15.119:3300/0 conn(0x7fc8e00b2010 msgr2=0x7fc8e00b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8dffff700 1 --2- 172.21.15.119:0/3907001019 >> v2:172.21.15.119:3300/0 conn(0x7fc8e00b2010 0x7fc8e00b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8dffff700 1 -- 172.21.15.119:0/3907001019 >> v1:172.21.15.92:6789/0 conn(0x7fc8e0153a80 legacy=0x7fc8e015a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8dffff700 1 -- 172.21.15.119:0/3907001019 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8e015aed0 con 0x7fc8e0153e50 2023-06-23T01:31:21.855 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8e56b2700 1 --2- 172.21.15.119:0/3907001019 >> v2:172.21.15.119:3300/0 conn(0x7fc8e00b2010 0x7fc8e00b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:21.856 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8dffff700 1 --2- 172.21.15.119:0/3907001019 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153e50 0x7fc8e00d0590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc0098e0 tx=0x7fc8cc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9460b0825e4830f7 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:21.856 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8df7fe700 1 -- 172.21.15.119:0/3907001019 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc00b640 con 0x7fc8e0153e50 2023-06-23T01:31:21.856 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8df7fe700 1 -- 172.21.15.119:0/3907001019 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8cc00b7a0 con 0x7fc8e0153e50 2023-06-23T01:31:21.856 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.852+0000 7fc8df7fe700 1 -- 172.21.15.119:0/3907001019 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc00bb40 con 0x7fc8e0153e50 2023-06-23T01:31:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.853+0000 7fc8e7115700 1 -- 172.21.15.119:0/3907001019 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153e50 msgr2=0x7fc8e00d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.853+0000 7fc8e7115700 1 --2- 172.21.15.119:0/3907001019 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153e50 0x7fc8e00d0590 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc0098e0 tx=0x7fc8cc0091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.853+0000 7fc8e7115700 1 -- 172.21.15.119:0/3907001019 shutdown_connections 2023-06-23T01:31:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.853+0000 7fc8e7115700 1 --2- 172.21.15.119:0/3907001019 >> v2:172.21.15.119:3300/0 conn(0x7fc8e00b2010 0x7fc8e00b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.858 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.853+0000 7fc8e7115700 1 --2- 172.21.15.119:0/3907001019 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153e50 0x7fc8e00d0590 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.853+0000 7fc8e7115700 1 -- 172.21.15.119:0/3907001019 >> 172.21.15.119:0/3907001019 conn(0x7fc8e0070290 msgr2=0x7fc8e0070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.853+0000 7fc8e7115700 1 -- 172.21.15.119:0/3907001019 shutdown_connections 2023-06-23T01:31:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.853+0000 7fc8e7115700 1 -- 172.21.15.119:0/3907001019 wait complete. 2023-06-23T01:31:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.854+0000 7fc8e7115700 1 Processor -- start 2023-06-23T01:31:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.854+0000 7fc8e7115700 1 -- start start 2023-06-23T01:31:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e7115700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00b2010 0x7fc8e0151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e7115700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153a80 0x7fc8e00cfd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e7115700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e0153e50 0x7fc8e00d02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e7115700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e0158750 con 0x7fc8e0153e50 2023-06-23T01:31:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e7115700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e016a6c0 con 0x7fc8e00b2010 2023-06-23T01:31:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e7115700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8e016a390 con 0x7fc8e0153a80 2023-06-23T01:31:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153a80 0x7fc8e00cfd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e4eb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00b2010 0x7fc8e0151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153a80 0x7fc8e00cfd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:37016/0 (socket says 172.21.15.119:37016) 2023-06-23T01:31:21.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e4eb1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00b2010 0x7fc8e0151d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51416/0 (socket says 172.21.15.119:51416) 2023-06-23T01:31:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 -- 172.21.15.119:0/3769686025 learned_addr learned my addr 172.21.15.119:0/3769686025 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e56b2700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e0153e50 0x7fc8e00d02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 -- 172.21.15.119:0/3769686025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00b2010 msgr2=0x7fc8e0151d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.861 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00b2010 0x7fc8e0151d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 -- 172.21.15.119:0/3769686025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e0153e50 msgr2=0x7fc8e00d02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e0153e50 0x7fc8e00d02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 -- 172.21.15.119:0/3769686025 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d8009ae0 con 0x7fc8e0153a80 2023-06-23T01:31:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8e4eb1700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00b2010 0x7fc8e0151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.855+0000 7fc8dffff700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153a80 0x7fc8e00cfd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009f80 tx=0x7fc8cc017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.856+0000 7fc8dcff9700 1 -- 172.21.15.119:0/3769686025 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc00bb20 con 0x7fc8e0153a80 2023-06-23T01:31:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.856+0000 7fc8dcff9700 1 -- 172.21.15.119:0/3769686025 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8cc000c90 con 0x7fc8e0153a80 2023-06-23T01:31:21.862 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.856+0000 7fc8dcff9700 1 -- 172.21.15.119:0/3769686025 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc011650 con 0x7fc8e0153a80 2023-06-23T01:31:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.856+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8cc009420 con 0x7fc8e0153a80 2023-06-23T01:31:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.856+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8e0158470 con 0x7fc8e0153a80 2023-06-23T01:31:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.856+0000 7fc8dcff9700 1 -- 172.21.15.119:0/3769686025 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc8cc011a40 con 0x7fc8e0153a80 2023-06-23T01:31:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.856+0000 7fc8dcff9700 1 -- 172.21.15.119:0/3769686025 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7fc8cc02c9e0 con 0x7fc8e0153a80 2023-06-23T01:31:21.863 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.856+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8e004c1b0 con 0x7fc8e0153a80 2023-06-23T01:31:21.864 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.859+0000 7fc8dcff9700 1 -- 172.21.15.119:0/3769686025 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fc8cc028070 con 0x7fc8e0153a80 2023-06-23T01:31:21.943 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.942+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "a2e05a19-0c70-4cb4-aacd-3381219435b8", "id": 1} v 0) v1 -- 0x7fc8e016a800 con 0x7fc8e0153a80 2023-06-23T01:31:21.950 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.949+0000 7fc8dcff9700 1 -- 172.21.15.119:0/3769686025 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a2e05a19-0c70-4cb4-aacd-3381219435b8", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc8cc015020 con 0x7fc8e0153a80 2023-06-23T01:31:21.950 INFO:teuthology.orchestra.run.smithi119.stdout:1 2023-06-23T01:31:21.951 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153a80 msgr2=0x7fc8e00cfd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:21.951 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153a80 0x7fc8e00cfd90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009f80 tx=0x7fc8cc017000 comp rx=0 tx=0).stop 2023-06-23T01:31:21.952 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 shutdown_connections 2023-06-23T01:31:21.952 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc8e0153e50 0x7fc8e00d02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.952 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc8e00b2010 0x7fc8e0151d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.952 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 --2- 172.21.15.119:0/3769686025 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc8e0153a80 0x7fc8e00cfd90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:21.952 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 >> 172.21.15.119:0/3769686025 conn(0x7fc8e0070290 msgr2=0x7fc8e00cf160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:21.952 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 shutdown_connections 2023-06-23T01:31:21.952 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:21.950+0000 7fc8e7115700 1 -- 172.21.15.119:0/3769686025 wait complete. 2023-06-23T01:31:21.964 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 20f85617-68ff-491f-a436-aa1ad1979595 2 2023-06-23T01:31:22.081 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.080+0000 7f14ac7b4700 1 Processor -- start 2023-06-23T01:31:22.081 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.080+0000 7f14ac7b4700 1 -- start start 2023-06-23T01:31:22.082 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14ac7b4700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f14a40b41d0 0x7f14a40b0760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.082 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14ac7b4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a40ce0b0 con 0x7f14a40aebc0 2023-06-23T01:31:22.082 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14ac7b4700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14a40cd4a0 con 0x7f14a40b3e00 2023-06-23T01:31:22.082 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14ac7b4700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14a40cd780 con 0x7f14a40b41d0 2023-06-23T01:31:22.082 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14aa550700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f14a40b3e00 0x7f14a40ae4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54658/0 (socket says 172.21.15.119:54658) 2023-06-23T01:31:22.082 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14aa550700 1 -- 172.21.15.119:0/451292700 learned_addr learned my addr 172.21.15.119:0/451292700 (peer_addr_for_me v1:172.21.15.119:0/0) 2023-06-23T01:31:22.083 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14a9d4f700 1 --2- 172.21.15.119:0/451292700 >> v2:172.21.15.92:3300/0 conn(0x7f14a40b41d0 0x7f14a40b0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.084 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14a954e700 1 -- 172.21.15.119:0/451292700 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1296612737 0 0) 0x7f14a40cd4a0 con 0x7f14a40b3e00 2023-06-23T01:31:22.084 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14a954e700 1 -- 172.21.15.119:0/451292700 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1490003570 con 0x7f14a40b3e00 2023-06-23T01:31:22.084 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14a9d4f700 1 -- 172.21.15.119:0/451292700 >> v1:172.21.15.119:6789/0 conn(0x7f14a40b3e00 legacy=0x7f14a40ae4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.084 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14a9d4f700 1 -- 172.21.15.119:0/451292700 >> v1:172.21.15.92:6789/0 conn(0x7f14a40aebc0 legacy=0x7f14a40b1340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.084 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14a954e700 1 -- 172.21.15.119:0/451292700 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 355049096 0 0) 0x7f14a40ce0b0 con 0x7f14a40aebc0 2023-06-23T01:31:22.085 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14a9d4f700 1 -- 172.21.15.119:0/451292700 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14a40cda60 con 0x7f14a40b41d0 2023-06-23T01:31:22.085 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.081+0000 7f14a9d4f700 1 --2- 172.21.15.119:0/451292700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 0x7f14a40b0760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14940098e0 tx=0x7f14940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=618f639c136d4b69 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.085 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14a954e700 1 -- 172.21.15.119:0/451292700 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149400b6d0 con 0x7f14a40b41d0 2023-06-23T01:31:22.085 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14a954e700 1 -- 172.21.15.119:0/451292700 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149400b830 con 0x7f14a40b41d0 2023-06-23T01:31:22.085 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14a954e700 1 -- 172.21.15.119:0/451292700 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149400bbd0 con 0x7f14a40b41d0 2023-06-23T01:31:22.085 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14ac7b4700 1 -- 172.21.15.119:0/451292700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 msgr2=0x7f14a40b0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.085 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14ac7b4700 1 --2- 172.21.15.119:0/451292700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 0x7f14a40b0760 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14940098e0 tx=0x7f14940091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.086 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14ac7b4700 1 -- 172.21.15.119:0/451292700 shutdown_connections 2023-06-23T01:31:22.086 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14ac7b4700 1 --2- 172.21.15.119:0/451292700 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 0x7f14a40b0760 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.086 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14ac7b4700 1 -- 172.21.15.119:0/451292700 >> 172.21.15.119:0/451292700 conn(0x7f14a4067bf0 msgr2=0x7f14a40ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.086 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14ac7b4700 1 -- 172.21.15.119:0/451292700 shutdown_connections 2023-06-23T01:31:22.086 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.082+0000 7f14ac7b4700 1 -- 172.21.15.119:0/451292700 wait complete. 2023-06-23T01:31:22.088 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14ac7b4700 1 Processor -- start 2023-06-23T01:31:22.088 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14ac7b4700 1 -- start start 2023-06-23T01:31:22.088 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14ac7b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14a40aebc0 0x7f14a41498b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.088 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14ac7b4700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14a40b3e00 0x7f14a4167b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.089 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14ac7b4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 0x7f14a415d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.089 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14ac7b4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a415c480 con 0x7f14a40b41d0 2023-06-23T01:31:22.089 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14ac7b4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14a415c6d0 con 0x7f14a40aebc0 2023-06-23T01:31:22.089 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14ac7b4700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14a415c9b0 con 0x7f14a40b3e00 2023-06-23T01:31:22.089 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14aa550700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14a40aebc0 0x7f14a41498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.089 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14a9d4f700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14a40b3e00 0x7f14a4167b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.089 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14a9d4f700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14a40b3e00 0x7f14a4167b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:37028/0 (socket says 172.21.15.119:37028) 2023-06-23T01:31:22.089 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14a9d4f700 1 -- 172.21.15.119:0/3170879503 learned_addr learned my addr 172.21.15.119:0/3170879503 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:22.090 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.084+0000 7f14aa550700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14a40aebc0 0x7f14a41498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51422/0 (socket says 172.21.15.119:51422) 2023-06-23T01:31:22.090 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14aad51700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 0x7f14a415d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.090 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14a9d4f700 1 -- 172.21.15.119:0/3170879503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14a40aebc0 msgr2=0x7f14a41498b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.090 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14a9d4f700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14a40aebc0 0x7f14a41498b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.090 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14a9d4f700 1 -- 172.21.15.119:0/3170879503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 msgr2=0x7f14a415d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.090 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14a9d4f700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 0x7f14a415d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.090 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14a9d4f700 1 -- 172.21.15.119:0/3170879503 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1494009420 con 0x7f14a40b3e00 2023-06-23T01:31:22.091 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14aa550700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14a40aebc0 0x7f14a41498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:22.091 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14a9d4f700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14a40b3e00 0x7f14a4167b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14940093f0 tx=0x7f1494017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.091 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f149a7fc700 1 -- 172.21.15.119:0/3170879503 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149401f070 con 0x7f14a40b3e00 2023-06-23T01:31:22.091 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f149a7fc700 1 -- 172.21.15.119:0/3170879503 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1494000bc0 con 0x7f14a40b3e00 2023-06-23T01:31:22.091 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14a415cc90 con 0x7f14a40b3e00 2023-06-23T01:31:22.091 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f149a7fc700 1 -- 172.21.15.119:0/3170879503 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149400bbd0 con 0x7f14a40b3e00 2023-06-23T01:31:22.091 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14a4149df0 con 0x7f14a40b3e00 2023-06-23T01:31:22.091 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f149a7fc700 1 -- 172.21.15.119:0/3170879503 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1494023070 con 0x7f14a40b3e00 2023-06-23T01:31:22.092 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.085+0000 7f149a7fc700 1 -- 172.21.15.119:0/3170879503 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f1494011850 con 0x7f14a40b3e00 2023-06-23T01:31:22.092 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.086+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14a404c1b0 con 0x7f14a40b3e00 2023-06-23T01:31:22.092 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:22.090+0000 7f39c8cc9180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:31:22.092 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.088+0000 7f149a7fc700 1 -- 172.21.15.119:0/3170879503 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f149402a070 con 0x7f14a40b3e00 2023-06-23T01:31:22.137 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:22.136+0000 7fcd5122f180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-06-23T01:31:22.173 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.171+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "20f85617-68ff-491f-a436-aa1ad1979595", "id": 2} v 0) v1 -- 0x7f14a415caf0 con 0x7f14a40b3e00 2023-06-23T01:31:22.180 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.179+0000 7f149a7fc700 1 -- 172.21.15.119:0/3170879503 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "20f85617-68ff-491f-a436-aa1ad1979595", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f14a415caf0 con 0x7f14a40b3e00 2023-06-23T01:31:22.181 INFO:teuthology.orchestra.run.smithi119.stdout:2 2023-06-23T01:31:22.181 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.180+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14a40b3e00 msgr2=0x7f14a4167b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.180+0000 7f14ac7b4700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14a40b3e00 0x7f14a4167b90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f14940093f0 tx=0x7f1494017000 comp rx=0 tx=0).stop 2023-06-23T01:31:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.180+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 shutdown_connections 2023-06-23T01:31:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.180+0000 7f14ac7b4700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14a40b41d0 0x7f14a415d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.180+0000 7f14ac7b4700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14a40aebc0 0x7f14a41498b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.180+0000 7f14ac7b4700 1 --2- 172.21.15.119:0/3170879503 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14a40b3e00 0x7f14a4167b90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.182 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.180+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 >> 172.21.15.119:0/3170879503 conn(0x7f14a4067bf0 msgr2=0x7f14a40ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.181+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 shutdown_connections 2023-06-23T01:31:22.183 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.181+0000 7f14ac7b4700 1 -- 172.21.15.119:0/3170879503 wait complete. 2023-06-23T01:31:22.193 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 3e5b05f4-6b77-4612-9720-38ece0b1388b 3 2023-06-23T01:31:22.224 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:22.222+0000 7f39c8cc9180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:31:22.277 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:22.275+0000 7fcd5122f180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-06-23T01:31:22.297 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:22.295+0000 7f39c8cc9180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:31:22.310 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.309+0000 7f3af5298700 1 Processor -- start 2023-06-23T01:31:22.311 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.309+0000 7f3af5298700 1 -- start start 2023-06-23T01:31:22.312 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3af5298700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3af00af930 0x7f3af01641b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.312 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3af5298700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af00aaf20 con 0x7f3af00aee50 2023-06-23T01:31:22.312 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3af5298700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af00ab4e0 con 0x7f3af00ac980 2023-06-23T01:31:22.312 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3af5298700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af00699f0 con 0x7f3af00af930 2023-06-23T01:31:22.313 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3aee7fc700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3af00ac980 0x7f3af00af220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:54662/0 (socket says 172.21.15.119:54662) 2023-06-23T01:31:22.313 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3aee7fc700 1 -- 172.21.15.119:0/28455555 learned_addr learned my addr 172.21.15.119:0/28455555 (peer_addr_for_me v1:172.21.15.119:0/0) 2023-06-23T01:31:22.313 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3aef7fe700 1 --2- 172.21.15.119:0/28455555 >> v2:172.21.15.119:3300/0 conn(0x7f3af00af930 0x7f3af01641b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.313 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3aedffb700 1 -- 172.21.15.119:0/28455555 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277306880 0 0) 0x7f3af00ab4e0 con 0x7f3af00ac980 2023-06-23T01:31:22.313 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3aedffb700 1 -- 172.21.15.119:0/28455555 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad8003570 con 0x7f3af00ac980 2023-06-23T01:31:22.313 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3aef7fe700 1 -- 172.21.15.119:0/28455555 >> v1:172.21.15.119:6789/0 conn(0x7f3af00ac980 legacy=0x7f3af00af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.313 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3aef7fe700 1 -- 172.21.15.119:0/28455555 >> v1:172.21.15.92:6789/0 conn(0x7f3af00aee50 legacy=0x7f3af0068f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.313 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.310+0000 7f3aef7fe700 1 -- 172.21.15.119:0/28455555 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af0069cd0 con 0x7f3af00af930 2023-06-23T01:31:22.314 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3aef7fe700 1 --2- 172.21.15.119:0/28455555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00af930 0x7f3af01641b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009e60 tx=0x7f3ae8009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38d667aa2506f463 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.314 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3aedffb700 1 -- 172.21.15.119:0/28455555 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae800e070 con 0x7f3af00af930 2023-06-23T01:31:22.314 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3aedffb700 1 -- 172.21.15.119:0/28455555 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae800bb60 con 0x7f3af00af930 2023-06-23T01:31:22.314 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3aedffb700 1 -- 172.21.15.119:0/28455555 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae8012430 con 0x7f3af00af930 2023-06-23T01:31:22.316 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3af5298700 1 -- 172.21.15.119:0/28455555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00af930 msgr2=0x7f3af01641b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.317 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3af5298700 1 --2- 172.21.15.119:0/28455555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00af930 0x7f3af01641b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f3ae8009e60 tx=0x7f3ae8009780 comp rx=0 tx=0).stop 2023-06-23T01:31:22.317 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3af5298700 1 -- 172.21.15.119:0/28455555 shutdown_connections 2023-06-23T01:31:22.317 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3af5298700 1 --2- 172.21.15.119:0/28455555 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00af930 0x7f3af01641b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.317 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3af5298700 1 -- 172.21.15.119:0/28455555 >> 172.21.15.119:0/28455555 conn(0x7f3af0087f50 msgr2=0x7f3af0088350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.317 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3af5298700 1 -- 172.21.15.119:0/28455555 shutdown_connections 2023-06-23T01:31:22.317 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.311+0000 7f3af5298700 1 -- 172.21.15.119:0/28455555 wait complete. 2023-06-23T01:31:22.318 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3af5298700 1 Processor -- start 2023-06-23T01:31:22.318 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3af5298700 1 -- start start 2023-06-23T01:31:22.318 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3af5298700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00ac980 0x7f3af0097f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.318 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3af5298700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3af00aee50 0x7f3af014adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.318 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3af5298700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00af930 0x7f3af009b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.318 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3af5298700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0098500 con 0x7f3af00af930 2023-06-23T01:31:22.318 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3af5298700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af00987e0 con 0x7f3af00ac980 2023-06-23T01:31:22.318 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3af5298700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af0098ac0 con 0x7f3af00aee50 2023-06-23T01:31:22.319 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3aeeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00ac980 0x7f3af0097f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.319 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3aeeffd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00ac980 0x7f3af0097f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:51440/0 (socket says 172.21.15.119:51440) 2023-06-23T01:31:22.319 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3aeeffd700 1 -- 172.21.15.119:0/774909949 learned_addr learned my addr 172.21.15.119:0/774909949 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:22.319 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.313+0000 7f3aee7fc700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3af00aee50 0x7f3af014adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.319 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3aef7fe700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00af930 0x7f3af009b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.319 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3aeeffd700 1 -- 172.21.15.119:0/774909949 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3af00aee50 msgr2=0x7f3af014adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.319 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3aeeffd700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3af00aee50 0x7f3af014adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.319 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3aeeffd700 1 -- 172.21.15.119:0/774909949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00af930 msgr2=0x7f3af009b380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.320 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3aeeffd700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00af930 0x7f3af009b380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.320 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3aeeffd700 1 -- 172.21.15.119:0/774909949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae8009a20 con 0x7f3af00ac980 2023-06-23T01:31:22.320 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3aeeffd700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00ac980 0x7f3af0097f20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c590 tx=0x7f3ae400c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.320 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3ad6ffd700 1 -- 172.21.15.119:0/774909949 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4011590 con 0x7f3af00ac980 2023-06-23T01:31:22.320 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3ad6ffd700 1 -- 172.21.15.119:0/774909949 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae4011710 con 0x7f3af00ac980 2023-06-23T01:31:22.320 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af0098da0 con 0x7f3af00ac980 2023-06-23T01:31:22.320 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3ad6ffd700 1 -- 172.21.15.119:0/774909949 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4011ad0 con 0x7f3af00ac980 2023-06-23T01:31:22.321 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af014b300 con 0x7f3af00ac980 2023-06-23T01:31:22.321 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3ad6ffd700 1 -- 172.21.15.119:0/774909949 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f3ae401c6c0 con 0x7f3af00ac980 2023-06-23T01:31:22.321 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.314+0000 7f3ad6ffd700 1 -- 172.21.15.119:0/774909949 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7f3ae40128c0 con 0x7f3af00ac980 2023-06-23T01:31:22.321 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.315+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af006d590 con 0x7f3af00ac980 2023-06-23T01:31:22.321 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.317+0000 7f3ad6ffd700 1 -- 172.21.15.119:0/774909949 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f3ae4017070 con 0x7f3af00ac980 2023-06-23T01:31:22.355 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:22.354+0000 7fcd5122f180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-06-23T01:31:22.401 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.400+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "3e5b05f4-6b77-4612-9720-38ece0b1388b", "id": 3} v 0) v1 -- 0x7f3af00992c0 con 0x7f3af00ac980 2023-06-23T01:31:22.409 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.408+0000 7f3ad6ffd700 1 -- 172.21.15.119:0/774909949 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3e5b05f4-6b77-4612-9720-38ece0b1388b", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3af00992c0 con 0x7f3af00ac980 2023-06-23T01:31:22.410 INFO:teuthology.orchestra.run.smithi119.stdout:3 2023-06-23T01:31:22.411 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00ac980 msgr2=0x7f3af0097f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.411 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00ac980 0x7f3af0097f20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3ae400c590 tx=0x7f3ae400c8a0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.411 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 shutdown_connections 2023-06-23T01:31:22.411 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3af00af930 0x7f3af009b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.411 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3af00ac980 0x7f3af0097f20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.411 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 --2- 172.21.15.119:0/774909949 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3af00aee50 0x7f3af014adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.412 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 >> 172.21.15.119:0/774909949 conn(0x7f3af0087f50 msgr2=0x7f3af006b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.412 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 shutdown_connections 2023-06-23T01:31:22.412 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.409+0000 7f3af5298700 1 -- 172.21.15.119:0/774909949 wait complete. 2023-06-23T01:31:22.423 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 4d94d9c9-4288-477f-801f-c66683295335 4 2023-06-23T01:31:22.480 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:22.478+0000 7f39c8cc9180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:31:22.539 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.538+0000 7f1e1e304700 1 Processor -- start 2023-06-23T01:31:22.539 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.538+0000 7f1e1e304700 1 -- start start 2023-06-23T01:31:22.541 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.538+0000 7f1e1e304700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e180ccd20 0x7f1e180cd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.541 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.538+0000 7f1e1e304700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e181595b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.542 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.538+0000 7f1e1e304700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1e18159af0 0x7f1e18069530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.542 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.538+0000 7f1e1e304700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e18068520 con 0x7f1e18159af0 2023-06-23T01:31:22.542 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.538+0000 7f1e1e304700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e18068770 con 0x7f1e180ccd20 2023-06-23T01:31:22.542 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.538+0000 7f1e1e304700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e18068a50 con 0x7f1e180cf0e0 2023-06-23T01:31:22.542 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e181595b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.543 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e17fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e180ccd20 0x7f1e180cd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.543 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e181595b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:40236/0 (socket says 172.21.15.119:40236) 2023-06-23T01:31:22.543 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e17fff700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1e180ccd20 0x7f1e180cd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40058/0 (socket says 172.21.15.119:40058) 2023-06-23T01:31:22.543 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 -- 172.21.15.119:0/1438023082 learned_addr learned my addr 172.21.15.119:0/1438023082 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:22.543 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e1c8a1700 1 --2- 172.21.15.119:0/1438023082 >> v2:172.21.15.92:3300/0 conn(0x7f1e18159af0 0x7f1e18069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.543 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 -- 172.21.15.119:0/1438023082 >> v2:172.21.15.119:3300/0 conn(0x7f1e180ccd20 msgr2=0x7f1e180cd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:22.543 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 --2- 172.21.15.119:0/1438023082 >> v2:172.21.15.119:3300/0 conn(0x7f1e180ccd20 0x7f1e180cd0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.544 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 -- 172.21.15.119:0/1438023082 >> v2:172.21.15.92:3300/0 conn(0x7f1e18159af0 msgr2=0x7f1e18069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:22.544 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 --2- 172.21.15.119:0/1438023082 >> v2:172.21.15.92:3300/0 conn(0x7f1e18159af0 0x7f1e18069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.544 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 -- 172.21.15.119:0/1438023082 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e18068d30 con 0x7f1e180cf0e0 2023-06-23T01:31:22.544 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e1c8a1700 1 --2- 172.21.15.119:0/1438023082 >> v2:172.21.15.92:3300/0 conn(0x7f1e18159af0 0x7f1e18069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:31:22.544 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e17fff700 1 --2- 172.21.15.119:0/1438023082 >> v2:172.21.15.119:3300/0 conn(0x7f1e180ccd20 0x7f1e180cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:22.544 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e177fe700 1 --2- 172.21.15.119:0/1438023082 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e181595b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1e180cc030 tx=0x7f1e08009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d64aa048219a991c server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.544 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e16ffd700 1 -- 172.21.15.119:0/1438023082 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e08009e70 con 0x7f1e180cf0e0 2023-06-23T01:31:22.545 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e16ffd700 1 -- 172.21.15.119:0/1438023082 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0800b860 con 0x7f1e180cf0e0 2023-06-23T01:31:22.545 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.539+0000 7f1e16ffd700 1 -- 172.21.15.119:0/1438023082 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0800bc00 con 0x7f1e180cf0e0 2023-06-23T01:31:22.545 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 -- 172.21.15.119:0/1438023082 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 msgr2=0x7f1e181595b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.545 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 --2- 172.21.15.119:0/1438023082 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e181595b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1e180cc030 tx=0x7f1e08009250 comp rx=0 tx=0).stop 2023-06-23T01:31:22.545 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 -- 172.21.15.119:0/1438023082 shutdown_connections 2023-06-23T01:31:22.546 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 --2- 172.21.15.119:0/1438023082 >> v2:172.21.15.119:3300/0 conn(0x7f1e180ccd20 0x7f1e180cd0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.546 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 --2- 172.21.15.119:0/1438023082 >> v2:172.21.15.92:3300/0 conn(0x7f1e18159af0 0x7f1e18069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.546 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 --2- 172.21.15.119:0/1438023082 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e181595b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.546 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 -- 172.21.15.119:0/1438023082 >> 172.21.15.119:0/1438023082 conn(0x7f1e18067bf0 msgr2=0x7f1e180ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.547 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 -- 172.21.15.119:0/1438023082 shutdown_connections 2023-06-23T01:31:22.547 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.540+0000 7f1e1e304700 1 -- 172.21.15.119:0/1438023082 wait complete. 2023-06-23T01:31:22.550 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.541+0000 7f1e1e304700 1 Processor -- start 2023-06-23T01:31:22.551 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.541+0000 7f1e1e304700 1 -- start start 2023-06-23T01:31:22.551 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e1e304700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180ccd20 0x7f1e18155320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.551 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e1e304700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e18099970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.552 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e1e304700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e18159af0 0x7f1e18098180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.552 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e1e304700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180986c0 con 0x7f1e18159af0 2023-06-23T01:31:22.552 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e1e304700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180994e0 con 0x7f1e180ccd20 2023-06-23T01:31:22.552 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e1e304700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e180988d0 con 0x7f1e180cf0e0 2023-06-23T01:31:22.553 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e177fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e18099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.553 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e17fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180ccd20 0x7f1e18155320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.553 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e177fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e18099970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:40252/0 (socket says 172.21.15.119:40252) 2023-06-23T01:31:22.553 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e177fe700 1 -- 172.21.15.119:0/1331504669 learned_addr learned my addr 172.21.15.119:0/1331504669 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:22.553 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e17fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180ccd20 0x7f1e18155320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40072/0 (socket says 172.21.15.119:40072) 2023-06-23T01:31:22.553 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e1c8a1700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e18159af0 0x7f1e18098180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.553 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e17fff700 1 -- 172.21.15.119:0/1331504669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 msgr2=0x7f1e18099970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e17fff700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e18099970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e17fff700 1 -- 172.21.15.119:0/1331504669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e18159af0 msgr2=0x7f1e18098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e17fff700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e18159af0 0x7f1e18098180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e17fff700 1 -- 172.21.15.119:0/1331504669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e0c0093c0 con 0x7f1e180ccd20 2023-06-23T01:31:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e177fe700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e18099970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e17fff700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180ccd20 0x7f1e18155320 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00ea50 tx=0x7f1e0c009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.554 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1dfffff700 1 -- 172.21.15.119:0/1331504669 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c00ee90 con 0x7f1e180ccd20 2023-06-23T01:31:22.555 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1dfffff700 1 -- 172.21.15.119:0/1331504669 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e0c005a00 con 0x7f1e180ccd20 2023-06-23T01:31:22.555 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e080094f0 con 0x7f1e180ccd20 2023-06-23T01:31:22.555 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.542+0000 7f1dfffff700 1 -- 172.21.15.119:0/1331504669 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c005dc0 con 0x7f1e180ccd20 2023-06-23T01:31:22.555 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.543+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e18098e90 con 0x7f1e180ccd20 2023-06-23T01:31:22.555 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.543+0000 7f1dfffff700 1 -- 172.21.15.119:0/1331504669 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1e0c00cc80 con 0x7f1e180ccd20 2023-06-23T01:31:22.555 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.543+0000 7f1dfffff700 1 -- 172.21.15.119:0/1331504669 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f1e0c0066c0 con 0x7f1e180ccd20 2023-06-23T01:31:22.555 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.543+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e18099eb0 con 0x7f1e180ccd20 2023-06-23T01:31:22.556 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.547+0000 7f1dfffff700 1 -- 172.21.15.119:0/1331504669 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f1e0c03a6e0 con 0x7f1e180ccd20 2023-06-23T01:31:22.556 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:22.549+0000 7fcd5122f180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:31:22.633 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.631+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4d94d9c9-4288-477f-801f-c66683295335", "id": 4} v 0) v1 -- 0x7f1e180cc680 con 0x7f1e180ccd20 2023-06-23T01:31:22.639 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.638+0000 7f1dfffff700 1 -- 172.21.15.119:0/1331504669 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4d94d9c9-4288-477f-801f-c66683295335", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1e0c029070 con 0x7f1e180ccd20 2023-06-23T01:31:22.639 INFO:teuthology.orchestra.run.smithi119.stdout:4 2023-06-23T01:31:22.640 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180ccd20 msgr2=0x7f1e18155320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.641 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180ccd20 0x7f1e18155320 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00ea50 tx=0x7f1e0c009120 comp rx=0 tx=0).stop 2023-06-23T01:31:22.641 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 shutdown_connections 2023-06-23T01:31:22.641 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1e18159af0 0x7f1e18098180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.641 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1e180ccd20 0x7f1e18155320 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.642 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 --2- 172.21.15.119:0/1331504669 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1e180cf0e0 0x7f1e18099970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.642 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 >> 172.21.15.119:0/1331504669 conn(0x7f1e18067bf0 msgr2=0x7f1e181563d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.642 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 shutdown_connections 2023-06-23T01:31:22.643 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.639+0000 7f1e1e304700 1 -- 172.21.15.119:0/1331504669 wait complete. 2023-06-23T01:31:22.651 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new dc6b2f59-8af4-42b2-b7f9-bf36714c358f 5 2023-06-23T01:31:22.653 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:22.651+0000 7f39c8cc9180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:31:22.729 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:22.727+0000 7f39c8cc9180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:31:22.730 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:22.728+0000 7fcd5122f180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-06-23T01:31:22.765 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.764+0000 7fcafbf32700 1 Processor -- start 2023-06-23T01:31:22.766 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.765+0000 7fcafbf32700 1 -- start start 2023-06-23T01:31:22.767 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.765+0000 7fcafbf32700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40b0bb0 0x7fcaf40cd090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.767 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.765+0000 7fcafbf32700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf40c4000 con 0x7fcaf40c9320 2023-06-23T01:31:22.768 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.765+0000 7fcafbf32700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcaf4157da0 con 0x7fcaf40b07e0 2023-06-23T01:31:22.768 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.765+0000 7fcafbf32700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf40c33f0 con 0x7fcaf40b0bb0 2023-06-23T01:31:22.768 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.765+0000 7fcaf94cd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40b0bb0 0x7fcaf40cd090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.768 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.765+0000 7fcaf9cce700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fcaf40b07e0 0x7fcaf40b1270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57428/0 (socket says 172.21.15.119:57428) 2023-06-23T01:31:22.768 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.765+0000 7fcaf9cce700 1 -- 172.21.15.119:0/373878771 learned_addr learned my addr 172.21.15.119:0/373878771 (peer_addr_for_me v1:172.21.15.119:0/0) 2023-06-23T01:31:22.768 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.766+0000 7fcaf94cd700 1 -- 172.21.15.119:0/373878771 >> v1:172.21.15.119:6789/0 conn(0x7fcaf40b07e0 legacy=0x7fcaf40b1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.768 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.766+0000 7fcaf94cd700 1 -- 172.21.15.119:0/373878771 >> v1:172.21.15.92:6789/0 conn(0x7fcaf40c9320 legacy=0x7fcaf40c97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.769 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.766+0000 7fcaf94cd700 1 -- 172.21.15.119:0/373878771 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf40c36d0 con 0x7fcaf40b0bb0 2023-06-23T01:31:22.769 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.766+0000 7fcaf94cd700 1 --2- 172.21.15.119:0/373878771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40b0bb0 0x7fcaf40cd090 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0009910 tx=0x7fcaf00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be3cc79b17b915c7 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.769 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.766+0000 7fcaf8ccc700 1 -- 172.21.15.119:0/373878771 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf000b860 con 0x7fcaf40b0bb0 2023-06-23T01:31:22.769 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.766+0000 7fcaf8ccc700 1 -- 172.21.15.119:0/373878771 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf000b9c0 con 0x7fcaf40b0bb0 2023-06-23T01:31:22.769 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.766+0000 7fcaf8ccc700 1 -- 172.21.15.119:0/373878771 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf000bd60 con 0x7fcaf40b0bb0 2023-06-23T01:31:22.771 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.767+0000 7fcafbf32700 1 -- 172.21.15.119:0/373878771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40b0bb0 msgr2=0x7fcaf40cd090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.772 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.767+0000 7fcafbf32700 1 --2- 172.21.15.119:0/373878771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40b0bb0 0x7fcaf40cd090 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcaf0009910 tx=0x7fcaf00091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.772 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.767+0000 7fcafbf32700 1 -- 172.21.15.119:0/373878771 shutdown_connections 2023-06-23T01:31:22.772 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.767+0000 7fcafbf32700 1 --2- 172.21.15.119:0/373878771 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40b0bb0 0x7fcaf40cd090 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.772 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.767+0000 7fcafbf32700 1 -- 172.21.15.119:0/373878771 >> 172.21.15.119:0/373878771 conn(0x7fcaf4067bf0 msgr2=0x7fcaf40ce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.772 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.767+0000 7fcafbf32700 1 -- 172.21.15.119:0/373878771 shutdown_connections 2023-06-23T01:31:22.773 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.767+0000 7fcafbf32700 1 -- 172.21.15.119:0/373878771 wait complete. 2023-06-23T01:31:22.773 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.768+0000 7fcafbf32700 1 Processor -- start 2023-06-23T01:31:22.773 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.768+0000 7fcafbf32700 1 -- start start 2023-06-23T01:31:22.773 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafbf32700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf40b07e0 0x7fcaf41535e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.773 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafbf32700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcaf40b0bb0 0x7fcaf40b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.773 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafbf32700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40c9320 0x7fcaf40985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.773 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafbf32700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf40b2c60 con 0x7fcaf40b07e0 2023-06-23T01:31:22.774 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafbf32700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf409b4f0 con 0x7fcaf40b0bb0 2023-06-23T01:31:22.774 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafbf32700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcaf409b7d0 con 0x7fcaf40c9320 2023-06-23T01:31:22.774 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcaf94cd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcaf40b0bb0 0x7fcaf40b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.774 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafa4cf700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40c9320 0x7fcaf40985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.774 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcaf94cd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcaf40b0bb0 0x7fcaf40b20c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40084/0 (socket says 172.21.15.119:40084) 2023-06-23T01:31:22.774 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcaf94cd700 1 -- 172.21.15.119:0/3055633322 learned_addr learned my addr 172.21.15.119:0/3055633322 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:22.774 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcaf9cce700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf40b07e0 0x7fcaf41535e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.774 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafa4cf700 1 -- 172.21.15.119:0/3055633322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcaf40b0bb0 msgr2=0x7fcaf40b20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.775 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafa4cf700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcaf40b0bb0 0x7fcaf40b20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.775 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafa4cf700 1 -- 172.21.15.119:0/3055633322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf40b07e0 msgr2=0x7fcaf41535e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.775 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafa4cf700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf40b07e0 0x7fcaf41535e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.775 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafa4cf700 1 -- 172.21.15.119:0/3055633322 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcaf0009420 con 0x7fcaf40c9320 2023-06-23T01:31:22.775 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcaf94cd700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcaf40b0bb0 0x7fcaf40b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:22.775 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafa4cf700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40c9320 0x7fcaf40985a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00c830 tx=0x7fcaec001690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.775 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcae9ffb700 1 -- 172.21.15.119:0/3055633322 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec012630 con 0x7fcaf40c9320 2023-06-23T01:31:22.775 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcae9ffb700 1 -- 172.21.15.119:0/3055633322 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaec012790 con 0x7fcaf40c9320 2023-06-23T01:31:22.776 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcae9ffb700 1 -- 172.21.15.119:0/3055633322 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec012b50 con 0x7fcaf40c9320 2023-06-23T01:31:22.776 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcaf409bab0 con 0x7fcaf40c9320 2023-06-23T01:31:22.776 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.769+0000 7fcae9ffb700 1 -- 172.21.15.119:0/3055633322 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fcaec01b680 con 0x7fcaf40c9320 2023-06-23T01:31:22.776 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.770+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcaf4153b20 con 0x7fcaf40c9320 2023-06-23T01:31:22.776 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.770+0000 7fcae9ffb700 1 -- 172.21.15.119:0/3055633322 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7fcaec01bac0 con 0x7fcaf40c9320 2023-06-23T01:31:22.776 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.770+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcaf409bcf0 con 0x7fcaf40c9320 2023-06-23T01:31:22.777 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.772+0000 7fcae9ffb700 1 -- 172.21.15.119:0/3055633322 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fcaf409bcf0 con 0x7fcaf40c9320 2023-06-23T01:31:22.810 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:22.809+0000 7fcd5122f180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-06-23T01:31:22.853 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.852+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "dc6b2f59-8af4-42b2-b7f9-bf36714c358f", "id": 5} v 0) v1 -- 0x7fcaf404c1b0 con 0x7fcaf40c9320 2023-06-23T01:31:22.858 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.857+0000 7fcae9ffb700 1 -- 172.21.15.119:0/3055633322 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "dc6b2f59-8af4-42b2-b7f9-bf36714c358f", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcaec01bc80 con 0x7fcaf40c9320 2023-06-23T01:31:22.858 INFO:teuthology.orchestra.run.smithi119.stdout:5 2023-06-23T01:31:22.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40c9320 msgr2=0x7fcaf40985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40c9320 0x7fcaf40985a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00c830 tx=0x7fcaec001690 comp rx=0 tx=0).stop 2023-06-23T01:31:22.859 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 shutdown_connections 2023-06-23T01:31:22.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fcaf40c9320 0x7fcaf40985a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fcaf40b07e0 0x7fcaf41535e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 --2- 172.21.15.119:0/3055633322 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fcaf40b0bb0 0x7fcaf40b20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 >> 172.21.15.119:0/3055633322 conn(0x7fcaf4067bf0 msgr2=0x7fcaf4156ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 shutdown_connections 2023-06-23T01:31:22.860 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.858+0000 7fcafbf32700 1 -- 172.21.15.119:0/3055633322 wait complete. 2023-06-23T01:31:22.864 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:22.862+0000 7f39c8cc9180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:31:22.871 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new 2ac0eb61-39bf-4666-8c24-279bf8b9c51d 6 2023-06-23T01:31:22.951 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:22.949+0000 7fcd5122f180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-06-23T01:31:22.959 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:22.957+0000 7f39c8cc9180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:31:22.988 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.987+0000 7f377508c700 1 Processor -- start 2023-06-23T01:31:22.989 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.987+0000 7f377508c700 1 -- start start 2023-06-23T01:31:22.992 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f377508c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f3770153900 0x7f37700ce490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.992 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f377508c700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37700b2d80 con 0x7f37700cab80 2023-06-23T01:31:22.992 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f377508c700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3770159d30 con 0x7f3770153cd0 2023-06-23T01:31:22.992 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f377508c700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3770159a90 con 0x7f3770153900 2023-06-23T01:31:22.993 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376f7fe700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3770153cd0 0x7f377015d090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57434/0 (socket says 172.21.15.119:57434) 2023-06-23T01:31:22.993 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376f7fe700 1 -- 172.21.15.119:0/1312929973 learned_addr learned my addr 172.21.15.119:0/1312929973 (peer_addr_for_me v1:172.21.15.119:0/0) 2023-06-23T01:31:22.993 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376ffff700 1 --2- 172.21.15.119:0/1312929973 >> v2:172.21.15.92:3300/0 conn(0x7f3770153900 0x7f37700ce490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.993 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376effd700 1 -- 172.21.15.119:0/1312929973 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748509461 0 0) 0x7f3770159d30 con 0x7f3770153cd0 2023-06-23T01:31:22.993 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376effd700 1 -- 172.21.15.119:0/1312929973 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3758003570 con 0x7f3770153cd0 2023-06-23T01:31:22.993 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376effd700 1 -- 172.21.15.119:0/1312929973 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3519651621 0 0) 0x7f3768001490 con 0x7f37700cab80 2023-06-23T01:31:22.993 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376effd700 1 -- 172.21.15.119:0/1312929973 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3770159d30 con 0x7f37700cab80 2023-06-23T01:31:22.994 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376ffff700 1 -- 172.21.15.119:0/1312929973 >> v1:172.21.15.119:6789/0 conn(0x7f3770153cd0 legacy=0x7f377015d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.994 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376effd700 1 -- 172.21.15.119:0/1312929973 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2940872933 0 0) 0x7f3770159d30 con 0x7f37700cab80 2023-06-23T01:31:22.994 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.988+0000 7f376ffff700 1 -- 172.21.15.119:0/1312929973 >> v1:172.21.15.92:6789/0 conn(0x7f37700cab80 legacy=0x7f377015a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.994 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f376ffff700 1 -- 172.21.15.119:0/1312929973 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37700cb000 con 0x7f3770153900 2023-06-23T01:31:22.994 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f376ffff700 1 --2- 172.21.15.119:0/1312929973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3770153900 0x7f37700ce490 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3764009910 tx=0x7f37640091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18050bc9067d9d64 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:22.994 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f376effd700 1 -- 172.21.15.119:0/1312929973 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768001980 con 0x7f3770153900 2023-06-23T01:31:22.994 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f376effd700 1 -- 172.21.15.119:0/1312929973 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3764011420 con 0x7f3770153900 2023-06-23T01:31:22.995 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f376effd700 1 -- 172.21.15.119:0/1312929973 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37640117c0 con 0x7f3770153900 2023-06-23T01:31:22.995 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f377508c700 1 -- 172.21.15.119:0/1312929973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3770153900 msgr2=0x7f37700ce490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:22.995 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f377508c700 1 --2- 172.21.15.119:0/1312929973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3770153900 0x7f37700ce490 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3764009910 tx=0x7f37640091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.995 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f377508c700 1 -- 172.21.15.119:0/1312929973 shutdown_connections 2023-06-23T01:31:22.995 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f377508c700 1 --2- 172.21.15.119:0/1312929973 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3770153900 0x7f37700ce490 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:22.995 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.989+0000 7f377508c700 1 -- 172.21.15.119:0/1312929973 >> 172.21.15.119:0/1312929973 conn(0x7f3770070290 msgr2=0x7f3770070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:22.995 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.990+0000 7f377508c700 1 -- 172.21.15.119:0/1312929973 shutdown_connections 2023-06-23T01:31:22.995 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.990+0000 7f377508c700 1 -- 172.21.15.119:0/1312929973 wait complete. 2023-06-23T01:31:22.996 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.991+0000 7f377508c700 1 Processor -- start 2023-06-23T01:31:22.996 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.991+0000 7f377508c700 1 -- start start 2023-06-23T01:31:22.998 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.991+0000 7f377508c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37700cab80 0x7f37700b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.998 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377508c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3770153900 0x7f37700b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.998 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377508c700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3770153cd0 0x7f3770151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:22.998 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377508c700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37700c79e0 con 0x7f37700cab80 2023-06-23T01:31:22.998 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377508c700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37700b3740 con 0x7f3770153900 2023-06-23T01:31:22.999 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377508c700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37700c76b0 con 0x7f3770153cd0 2023-06-23T01:31:22.999 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f376f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3770153900 0x7f37700b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.999 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377488b700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3770153cd0 0x7f3770151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.999 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f376ffff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37700cab80 0x7f37700b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:22.999 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f376f7fe700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3770153900 0x7f37700b1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40088/0 (socket says 172.21.15.119:40088) 2023-06-23T01:31:22.999 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f376f7fe700 1 -- 172.21.15.119:0/3200082789 learned_addr learned my addr 172.21.15.119:0/3200082789 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:22.999 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377488b700 1 -- 172.21.15.119:0/3200082789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3770153900 msgr2=0x7f37700b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.000 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377488b700 1 --2- 172.21.15.119:0/3200082789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3770153900 0x7f37700b1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.000 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377488b700 1 -- 172.21.15.119:0/3200082789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37700cab80 msgr2=0x7f37700b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.000 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377488b700 1 --2- 172.21.15.119:0/3200082789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37700cab80 0x7f37700b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.000 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377488b700 1 -- 172.21.15.119:0/3200082789 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3764009420 con 0x7f3770153cd0 2023-06-23T01:31:23.000 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f376f7fe700 1 --2- 172.21.15.119:0/3200082789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3770153900 0x7f37700b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:23.000 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377488b700 1 --2- 172.21.15.119:0/3200082789 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3770153cd0 0x7f3770151000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f376800cbc0 tx=0x7f3768001610 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:23.000 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f3757fff700 1 -- 172.21.15.119:0/3200082789 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3768001ea0 con 0x7f3770153cd0 2023-06-23T01:31:23.001 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f3757fff700 1 -- 172.21.15.119:0/3200082789 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376800a7a0 con 0x7f3770153cd0 2023-06-23T01:31:23.001 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37700b3f80 con 0x7f3770153cd0 2023-06-23T01:31:23.001 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f3757fff700 1 -- 172.21.15.119:0/3200082789 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376800ab60 con 0x7f3770153cd0 2023-06-23T01:31:23.001 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37700b3ca0 con 0x7f3770153cd0 2023-06-23T01:31:23.001 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.992+0000 7f3757fff700 1 -- 172.21.15.119:0/3200082789 <== mon.2 v2:172.21.15.119:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f376801b400 con 0x7f3770153cd0 2023-06-23T01:31:23.001 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.993+0000 7f3757fff700 1 -- 172.21.15.119:0/3200082789 <== mon.2 v2:172.21.15.119:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f3768012400 con 0x7f3770153cd0 2023-06-23T01:31:23.001 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.994+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37700c7510 con 0x7f3770153cd0 2023-06-23T01:31:23.002 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:22.996+0000 7f3757fff700 1 -- 172.21.15.119:0/3200082789 <== mon.2 v2:172.21.15.119:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f37700c7510 con 0x7f3770153cd0 2023-06-23T01:31:23.083 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:23.082+0000 7fcd5122f180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-06-23T01:31:23.128 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.127+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2ac0eb61-39bf-4666-8c24-279bf8b9c51d", "id": 6} v 0) v1 -- 0x7f37700c7b90 con 0x7f3770153cd0 2023-06-23T01:31:23.134 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.133+0000 7f3757fff700 1 -- 172.21.15.119:0/3200082789 <== mon.2 v2:172.21.15.119:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2ac0eb61-39bf-4666-8c24-279bf8b9c51d", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f37700c7b90 con 0x7f3770153cd0 2023-06-23T01:31:23.135 INFO:teuthology.orchestra.run.smithi119.stdout:6 2023-06-23T01:31:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.134+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3770153cd0 msgr2=0x7f3770151000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.134+0000 7f377508c700 1 --2- 172.21.15.119:0/3200082789 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3770153cd0 0x7f3770151000 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f376800cbc0 tx=0x7f3768001610 comp rx=0 tx=0).stop 2023-06-23T01:31:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.134+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 shutdown_connections 2023-06-23T01:31:23.136 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.134+0000 7f377508c700 1 --2- 172.21.15.119:0/3200082789 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3770153cd0 0x7f3770151000 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.137 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.134+0000 7f377508c700 1 --2- 172.21.15.119:0/3200082789 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f37700cab80 0x7f37700b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.137 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.134+0000 7f377508c700 1 --2- 172.21.15.119:0/3200082789 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3770153900 0x7f37700b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.137 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.134+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 >> 172.21.15.119:0/3200082789 conn(0x7f3770070290 msgr2=0x7f3770152c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:23.137 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.135+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 shutdown_connections 2023-06-23T01:31:23.137 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.135+0000 7f377508c700 1 -- 172.21.15.119:0/3200082789 wait complete. 2023-06-23T01:31:23.147 DEBUG:teuthology.orchestra.run.smithi119:> sudo ceph --cluster ceph osd new c4ff77b0-23dc-477d-adaa-d7fd9108f5da 7 2023-06-23T01:31:23.261 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.259+0000 7fd84df8b700 1 Processor -- start 2023-06-23T01:31:23.261 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.259+0000 7fd84df8b700 1 -- start start 2023-06-23T01:31:23.263 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.260+0000 7fd84df8b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd8481570b0 0x7fd84815a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.263 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.260+0000 7fd84df8b700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8480cef00 con 0x7fd8480b2010 2023-06-23T01:31:23.263 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.260+0000 7fd84df8b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8480cec00 con 0x7fd848157480 2023-06-23T01:31:23.263 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.260+0000 7fd84df8b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd84815db50 con 0x7fd8481570b0 2023-06-23T01:31:23.264 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.260+0000 7fd8477fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fd8481570b0 0x7fd84815a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.264 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.260+0000 7fd846ffd700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fd848157480 0x7fd84815d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.119:57448/0 (socket says 172.21.15.119:57448) 2023-06-23T01:31:23.264 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.260+0000 7fd846ffd700 1 -- 172.21.15.119:0/719563582 learned_addr learned my addr 172.21.15.119:0/719563582 (peer_addr_for_me v1:172.21.15.119:0/0) 2023-06-23T01:31:23.265 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd8477fe700 1 -- 172.21.15.119:0/719563582 >> v1:172.21.15.119:6789/0 conn(0x7fd848157480 legacy=0x7fd84815d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.265 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd8477fe700 1 -- 172.21.15.119:0/719563582 >> v1:172.21.15.92:6789/0 conn(0x7fd8480b2010 legacy=0x7fd8480cf390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.265 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd8477fe700 1 -- 172.21.15.119:0/719563582 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd84815de30 con 0x7fd8481570b0 2023-06-23T01:31:23.265 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd8467fc700 1 -- 172.21.15.119:0/719563582 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2067228557 0 0) 0x7fd8480cec00 con 0x7fd848157480 2023-06-23T01:31:23.265 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd8477fe700 1 --2- 172.21.15.119:0/719563582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481570b0 0x7fd84815a8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd8380098e0 tx=0x7fd8380091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ef435d50a244208 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:23.265 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd8467fc700 1 -- 172.21.15.119:0/719563582 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd838009e80 con 0x7fd8481570b0 2023-06-23T01:31:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd8467fc700 1 -- 172.21.15.119:0/719563582 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd83800b860 con 0x7fd8481570b0 2023-06-23T01:31:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd8467fc700 1 -- 172.21.15.119:0/719563582 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd83800bbe0 con 0x7fd8481570b0 2023-06-23T01:31:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd84df8b700 1 -- 172.21.15.119:0/719563582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481570b0 msgr2=0x7fd84815a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd84df8b700 1 --2- 172.21.15.119:0/719563582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481570b0 0x7fd84815a8a0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd8380098e0 tx=0x7fd8380091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd84df8b700 1 -- 172.21.15.119:0/719563582 shutdown_connections 2023-06-23T01:31:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd84df8b700 1 --2- 172.21.15.119:0/719563582 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd8481570b0 0x7fd84815a8a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.266 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd84df8b700 1 -- 172.21.15.119:0/719563582 >> 172.21.15.119:0/719563582 conn(0x7fd848070290 msgr2=0x7fd848070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:23.267 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd84df8b700 1 -- 172.21.15.119:0/719563582 shutdown_connections 2023-06-23T01:31:23.267 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.261+0000 7fd84df8b700 1 -- 172.21.15.119:0/719563582 wait complete. 2023-06-23T01:31:23.269 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd84df8b700 1 Processor -- start 2023-06-23T01:31:23.269 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd84df8b700 1 -- start start 2023-06-23T01:31:23.269 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd84df8b700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8480b2010 0x7fd8480a6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.270 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd84df8b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8481570b0 0x7fd84815c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.270 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd84df8b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd848157480 0x7fd848152fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.270 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd84df8b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480a8070 con 0x7fd8481570b0 2023-06-23T01:31:23.270 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd84df8b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480a74f0 con 0x7fd848157480 2023-06-23T01:31:23.270 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd84df8b700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8480a77d0 con 0x7fd8480b2010 2023-06-23T01:31:23.270 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd847fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd848157480 0x7fd848152fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.270 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd8477fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8480b2010 0x7fd8480a6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd847fff700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd848157480 0x7fd848152fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.119:40102/0 (socket says 172.21.15.119:40102) 2023-06-23T01:31:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd8477fe700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8480b2010 0x7fd8480a6f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.119:40284/0 (socket says 172.21.15.119:40284) 2023-06-23T01:31:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd847fff700 1 -- 172.21.15.119:0/1419205353 learned_addr learned my addr 172.21.15.119:0/1419205353 (peer_addr_for_me v2:172.21.15.119:0/0) 2023-06-23T01:31:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.263+0000 7fd846ffd700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8481570b0 0x7fd84815c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd847fff700 1 -- 172.21.15.119:0/1419205353 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8480b2010 msgr2=0x7fd8480a6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd847fff700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8480b2010 0x7fd8480a6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.271 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd847fff700 1 -- 172.21.15.119:0/1419205353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8481570b0 msgr2=0x7fd84815c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.272 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd847fff700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8481570b0 0x7fd84815c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.272 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd847fff700 1 -- 172.21.15.119:0/1419205353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd838009420 con 0x7fd848157480 2023-06-23T01:31:23.272 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd8477fe700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8480b2010 0x7fd8480a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:23.272 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd847fff700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd848157480 0x7fd848152fe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd84000c8e0 tx=0x7fd840001710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:23.272 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd82f7fe700 1 -- 172.21.15.119:0/1419205353 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840012630 con 0x7fd848157480 2023-06-23T01:31:23.272 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd82f7fe700 1 -- 172.21.15.119:0/1419205353 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd840012790 con 0x7fd848157480 2023-06-23T01:31:23.273 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8480a7ab0 con 0x7fd848157480 2023-06-23T01:31:23.273 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd82f7fe700 1 -- 172.21.15.119:0/1419205353 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd840012b50 con 0x7fd848157480 2023-06-23T01:31:23.273 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd84815c670 con 0x7fd848157480 2023-06-23T01:31:23.273 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd82f7fe700 1 -- 172.21.15.119:0/1419205353 <== mon.1 v2:172.21.15.119:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fd84001b400 con 0x7fd848157480 2023-06-23T01:31:23.273 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.264+0000 7fd82f7fe700 1 -- 172.21.15.119:0/1419205353 <== mon.1 v2:172.21.15.119:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1526+0+0 (secure 0 0 0) 0x7fd84001a3a0 con 0x7fd848157480 2023-06-23T01:31:23.273 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.265+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd84804c1b0 con 0x7fd848157480 2023-06-23T01:31:23.273 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.267+0000 7fd82f7fe700 1 -- 172.21.15.119:0/1419205353 <== mon.1 v2:172.21.15.119:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fd84001b6b0 con 0x7fd848157480 2023-06-23T01:31:23.354 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.352+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c4ff77b0-23dc-477d-adaa-d7fd9108f5da", "id": 7} v 0) v1 -- 0x7fd848169b30 con 0x7fd848157480 2023-06-23T01:31:23.362 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.360+0000 7fd82f7fe700 1 -- 172.21.15.119:0/1419205353 <== mon.1 v2:172.21.15.119:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c4ff77b0-23dc-477d-adaa-d7fd9108f5da", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd84001b850 con 0x7fd848157480 2023-06-23T01:31:23.363 INFO:teuthology.orchestra.run.smithi119.stdout:7 2023-06-23T01:31:23.364 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.361+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd848157480 msgr2=0x7fd848152fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.364 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.362+0000 7fd84df8b700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd848157480 0x7fd848152fe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fd84000c8e0 tx=0x7fd840001710 comp rx=0 tx=0).stop 2023-06-23T01:31:23.364 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.362+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 shutdown_connections 2023-06-23T01:31:23.365 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.362+0000 7fd84df8b700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fd848157480 0x7fd848152fe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.365 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.362+0000 7fd84df8b700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fd8480b2010 0x7fd8480a6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.366 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.362+0000 7fd84df8b700 1 --2- 172.21.15.119:0/1419205353 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fd8481570b0 0x7fd84815c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.366 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.362+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 >> 172.21.15.119:0/1419205353 conn(0x7fd848070290 msgr2=0x7fd8480b0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:23.366 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.362+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 shutdown_connections 2023-06-23T01:31:23.366 INFO:teuthology.orchestra.run.smithi119.stderr:2023-06-23T01:31:23.362+0000 7fd84df8b700 1 -- 172.21.15.119:0/1419205353 wait complete. 2023-06-23T01:31:23.375 INFO:tasks.ceph.osd.0:Restarting daemon 2023-06-23T01:31:23.376 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-06-23T01:31:23.379 INFO:tasks.ceph.osd.0:Started 2023-06-23T01:31:23.380 INFO:tasks.ceph.osd.1:Restarting daemon 2023-06-23T01:31:23.381 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-06-23T01:31:23.383 INFO:tasks.ceph.osd.1:Started 2023-06-23T01:31:23.383 INFO:tasks.ceph.osd.2:Restarting daemon 2023-06-23T01:31:23.384 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-06-23T01:31:23.386 INFO:tasks.ceph.osd.2:Started 2023-06-23T01:31:23.387 INFO:tasks.ceph.osd.3:Restarting daemon 2023-06-23T01:31:23.387 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-06-23T01:31:23.390 INFO:tasks.ceph.osd.3:Started 2023-06-23T01:31:23.391 INFO:tasks.ceph.osd.4:Restarting daemon 2023-06-23T01:31:23.391 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-06-23T01:31:23.393 INFO:tasks.ceph.osd.4:Started 2023-06-23T01:31:23.394 INFO:tasks.ceph.osd.5:Restarting daemon 2023-06-23T01:31:23.394 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-06-23T01:31:23.396 INFO:tasks.ceph.osd.5:Started 2023-06-23T01:31:23.396 INFO:tasks.ceph.osd.6:Restarting daemon 2023-06-23T01:31:23.396 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-06-23T01:31:23.398 INFO:tasks.ceph.osd.6:Started 2023-06-23T01:31:23.399 INFO:tasks.ceph.osd.7:Restarting daemon 2023-06-23T01:31:23.399 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-06-23T01:31:23.401 INFO:tasks.ceph.osd.7:Started 2023-06-23T01:31:23.402 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:31:23.502 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:23.500+0000 7f39c8cc9180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:31:23.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.540+0000 7fac9b33b700 1 Processor -- start 2023-06-23T01:31:23.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.540+0000 7fac9b33b700 1 -- start start 2023-06-23T01:31:23.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac9b33b700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac940ad770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac9b33b700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fac94156690 0x7fac940adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac9b33b700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac940aecc0 con 0x7fac940b08c0 2023-06-23T01:31:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac9b33b700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac940b1e70 con 0x7fac94156690 2023-06-23T01:31:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac9b33b700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac940ad220 con 0x7fac940b04f0 2023-06-23T01:31:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac9a339700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac940ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac9ab3a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fac94156690 0x7fac940adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac99b38700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fac940b08c0 0x7fac94155f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.92:40026/0 (socket says 172.21.15.92:40026) 2023-06-23T01:31:23.545 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.541+0000 7fac99b38700 1 -- 172.21.15.92:0/1120169440 learned_addr learned my addr 172.21.15.92:0/1120169440 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac9a339700 1 -- 172.21.15.92:0/1120169440 >> v2:172.21.15.119:3300/0 conn(0x7fac94156690 msgr2=0x7fac940adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac9a339700 1 --2- 172.21.15.92:0/1120169440 >> v2:172.21.15.119:3300/0 conn(0x7fac94156690 0x7fac940adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac9a339700 1 -- 172.21.15.92:0/1120169440 >> v1:172.21.15.119:6789/0 conn(0x7fac940b08c0 legacy=0x7fac94155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac99337700 1 -- 172.21.15.92:0/1120169440 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820113572 0 0) 0x7fac940aecc0 con 0x7fac940b08c0 2023-06-23T01:31:23.546 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac9a339700 1 -- 172.21.15.92:0/1120169440 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac94156ce0 con 0x7fac940b04f0 2023-06-23T01:31:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac9ab3a700 1 --2- 172.21.15.92:0/1120169440 >> v2:172.21.15.119:3300/0 conn(0x7fac94156690 0x7fac940adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac9a339700 1 --2- 172.21.15.92:0/1120169440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac940ad770 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fac84009900 tx=0x7fac84009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b71645b5d21f119a server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac99337700 1 -- 172.21.15.92:0/1120169440 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84005c10 con 0x7fac940b04f0 2023-06-23T01:31:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac99337700 1 -- 172.21.15.92:0/1120169440 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac84005d70 con 0x7fac940b04f0 2023-06-23T01:31:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac99337700 1 -- 172.21.15.92:0/1120169440 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac84006260 con 0x7fac940b04f0 2023-06-23T01:31:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac9b33b700 1 -- 172.21.15.92:0/1120169440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 msgr2=0x7fac940ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.542+0000 7fac9b33b700 1 --2- 172.21.15.92:0/1120169440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac940ad770 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fac84009900 tx=0x7fac84009120 comp rx=0 tx=0).stop 2023-06-23T01:31:23.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.543+0000 7fac9b33b700 1 -- 172.21.15.92:0/1120169440 shutdown_connections 2023-06-23T01:31:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.543+0000 7fac9b33b700 1 --2- 172.21.15.92:0/1120169440 >> v2:172.21.15.119:3300/0 conn(0x7fac94156690 0x7fac940adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.543+0000 7fac9b33b700 1 --2- 172.21.15.92:0/1120169440 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac940ad770 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.543+0000 7fac9b33b700 1 -- 172.21.15.92:0/1120169440 >> 172.21.15.92:0/1120169440 conn(0x7fac94067410 msgr2=0x7fac9406b760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.543+0000 7fac9b33b700 1 -- 172.21.15.92:0/1120169440 shutdown_connections 2023-06-23T01:31:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.543+0000 7fac9b33b700 1 -- 172.21.15.92:0/1120169440 wait complete. 2023-06-23T01:31:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9b33b700 1 Processor -- start 2023-06-23T01:31:23.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9b33b700 1 -- start start 2023-06-23T01:31:23.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9b33b700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac94164c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9b33b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac940b08c0 0x7fac94165e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9b33b700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac94156690 0x7fac94154080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:23.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9b33b700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94167550 con 0x7fac940b08c0 2023-06-23T01:31:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9b33b700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac94166940 con 0x7fac94156690 2023-06-23T01:31:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9b33b700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac94166c20 con 0x7fac940b04f0 2023-06-23T01:31:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac99b38700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac940b08c0 0x7fac94165e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9a339700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac94164c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac99b38700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac940b08c0 0x7fac94165e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:41178/0 (socket says 172.21.15.92:41178) 2023-06-23T01:31:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac99b38700 1 -- 172.21.15.92:0/1491055026 learned_addr learned my addr 172.21.15.92:0/1491055026 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:23.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.544+0000 7fac9ab3a700 1 --2- 172.21.15.92:0/1491055026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac94156690 0x7fac94154080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac99b38700 1 -- 172.21.15.92:0/1491055026 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 msgr2=0x7fac94164c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac99b38700 1 --2- 172.21.15.92:0/1491055026 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac94164c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac99b38700 1 -- 172.21.15.92:0/1491055026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac94156690 msgr2=0x7fac94154080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac99b38700 1 --2- 172.21.15.92:0/1491055026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac94156690 0x7fac94154080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac99b38700 1 -- 172.21.15.92:0/1491055026 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac8c009a20 con 0x7fac940b08c0 2023-06-23T01:31:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac99b38700 1 --2- 172.21.15.92:0/1491055026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac940b08c0 0x7fac94165e90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fac9000c7e0 tx=0x7fac90007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:23.552 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac8a7fc700 1 -- 172.21.15.92:0/1491055026 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90005b30 con 0x7fac940b08c0 2023-06-23T01:31:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac8a7fc700 1 -- 172.21.15.92:0/1491055026 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac90005c90 con 0x7fac940b08c0 2023-06-23T01:31:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac8a7fc700 1 -- 172.21.15.92:0/1491055026 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac90007300 con 0x7fac940b08c0 2023-06-23T01:31:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac840093c0 con 0x7fac940b08c0 2023-06-23T01:31:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.545+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac941671e0 con 0x7fac940b08c0 2023-06-23T01:31:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.546+0000 7fac8a7fc700 1 -- 172.21.15.92:0/1491055026 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fac9001c440 con 0x7fac940b08c0 2023-06-23T01:31:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.546+0000 7fac8a7fc700 1 -- 172.21.15.92:0/1491055026 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7fac90005e00 con 0x7fac940b08c0 2023-06-23T01:31:23.553 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.546+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac941676c0 con 0x7fac940b08c0 2023-06-23T01:31:23.554 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.550+0000 7fac8a7fc700 1 -- 172.21.15.92:0/1491055026 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7fac90017070 con 0x7fac940b08c0 2023-06-23T01:31:23.585 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:23.583+0000 7f39c8cc9180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:31:23.620 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.618+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac9406c950 con 0x7fac940b08c0 2023-06-23T01:31:23.621 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.619+0000 7fac8a7fc700 1 -- 172.21.15.92:0/1491055026 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7fac9001c830 con 0x7fac940b08c0 2023-06-23T01:31:23.621 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:31:23.621 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":10,"fsid":"c94811cf-f7e5-4c57-b505-a983532f8da8","created":"2023-06-23T01:31:20.907484+0000","modified":"2023-06-23T01:31:23.354162+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:31:23.354162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4bcf4cdc-7c98-4a4a-bc56-133f22f4d802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a2e05a19-0c70-4cb4-aacd-3381219435b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"20f85617-68ff-491f-a436-aa1ad1979595","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3e5b05f4-6b77-4612-9720-38ece0b1388b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d94d9c9-4288-477f-801f-c66683295335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6b2f59-8af4-42b2-b7f9-bf36714c358f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2ac0eb61-39bf-4666-8c24-279bf8b9c51d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c4ff77b0-23dc-477d-adaa-d7fd9108f5da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:31:23.623 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac940b08c0 msgr2=0x7fac94165e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:23.623 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 --2- 172.21.15.92:0/1491055026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac940b08c0 0x7fac94165e90 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fac9000c7e0 tx=0x7fac90007df0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.623 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 shutdown_connections 2023-06-23T01:31:23.623 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 --2- 172.21.15.92:0/1491055026 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fac94156690 0x7fac94154080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.623 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 --2- 172.21.15.92:0/1491055026 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fac940b04f0 0x7fac94164c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.623 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 --2- 172.21.15.92:0/1491055026 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fac940b08c0 0x7fac94165e90 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:23.624 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 >> 172.21.15.92:0/1491055026 conn(0x7fac94067410 msgr2=0x7fac94063340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:23.624 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 shutdown_connections 2023-06-23T01:31:23.624 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:23.620+0000 7fac9b33b700 1 -- 172.21.15.92:0/1491055026 wait complete. 2023-06-23T01:31:23.627 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:23.626+0000 7fcd5122f180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-06-23T01:31:23.634 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-06-23T01:31:23.635 INFO:tasks.ceph:Waiting for OSDs to come up 2023-06-23T01:31:23.665 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:23.663+0000 7f39c8cc9180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:31:23.704 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:23.703+0000 7fcd5122f180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-06-23T01:31:23.782 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:23.781+0000 7fcd5122f180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-06-23T01:31:23.985 INFO:tasks.ceph.osd.4.smithi119.stderr:2023-06-23T01:31:23.983+0000 7f5b094c3540 -1 Falling back to public interface 2023-06-23T01:31:23.985 INFO:tasks.ceph.osd.5.smithi119.stderr:2023-06-23T01:31:23.983+0000 7f6a9e10d540 -1 Falling back to public interface 2023-06-23T01:31:23.986 INFO:tasks.ceph.osd.4.smithi119.stderr:2023-06-23T01:31:23.983+0000 7f5b038ec700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6800/0: (98) Address already in use 2023-06-23T01:31:23.986 INFO:tasks.ceph.osd.5.smithi119.stderr:2023-06-23T01:31:23.983+0000 7f6a98536700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6801/0: (98) Address already in use 2023-06-23T01:31:23.987 INFO:tasks.ceph.osd.5.smithi119.stderr:2023-06-23T01:31:23.984+0000 7f6a98536700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6802/0: (98) Address already in use 2023-06-23T01:31:23.990 INFO:tasks.ceph.osd.2.smithi092.stderr:2023-06-23T01:31:23.988+0000 7fc09a699540 -1 Falling back to public interface 2023-06-23T01:31:23.993 INFO:tasks.ceph.osd.0.smithi092.stderr:2023-06-23T01:31:23.991+0000 7f0ec6f3b540 -1 Falling back to public interface 2023-06-23T01:31:23.999 INFO:tasks.ceph.osd.3.smithi092.stderr:2023-06-23T01:31:23.997+0000 7ff9622b8540 -1 Falling back to public interface 2023-06-23T01:31:23.999 INFO:tasks.ceph.osd.1.smithi092.stderr:2023-06-23T01:31:23.997+0000 7f6beab55540 -1 Falling back to public interface 2023-06-23T01:31:24.019 INFO:tasks.ceph.osd.6.smithi119.stderr:2023-06-23T01:31:24.018+0000 7ff6bc7aa540 -1 Falling back to public interface 2023-06-23T01:31:24.020 INFO:tasks.ceph.osd.7.smithi119.stderr:2023-06-23T01:31:24.018+0000 7fb2b31f5540 -1 Falling back to public interface 2023-06-23T01:31:24.020 INFO:tasks.ceph.osd.6.smithi119.stderr:2023-06-23T01:31:24.019+0000 7ff6b6bd3700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2023-06-23T01:31:24.233 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:24.231+0000 7f39c8cc9180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:31:24.371 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:24.369+0000 7f39c8cc9180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:31:24.373 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:24.372+0000 7fcd5122f180 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-06-23T01:31:24.441 DEBUG:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:31:24.460 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:24.458+0000 7f39c8cc9180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:31:24.510 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:24.509+0000 7fcd5122f180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-06-23T01:31:24.552 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.549+0000 7f9d590b4700 1 Processor -- start 2023-06-23T01:31:24.552 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.549+0000 7f9d590b4700 1 -- start start 2023-06-23T01:31:24.554 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.550+0000 7f9d590b4700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f9d540cafa0 0x7f9d540c8990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:24.554 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.550+0000 7f9d590b4700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540c8ed0 0x7f9d541574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:24.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.550+0000 7f9d590b4700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d540c9350 con 0x7f9d54158710 2023-06-23T01:31:24.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.550+0000 7f9d590b4700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d5415bef0 con 0x7f9d540cafa0 2023-06-23T01:31:24.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.550+0000 7f9d590b4700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5415b130 con 0x7f9d540c8ed0 2023-06-23T01:31:24.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.550+0000 7f9d5259c700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9d54158710 0x7f9d54156d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:44356/0 (socket says 172.21.15.92:44356) 2023-06-23T01:31:24.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.550+0000 7f9d5259c700 1 -- 172.21.15.92:0/1219014898 learned_addr learned my addr 172.21.15.92:0/1219014898 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:24.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d52d9d700 1 --2- 172.21.15.92:0/1219014898 >> v2:172.21.15.119:3300/0 conn(0x7f9d540cafa0 0x7f9d540c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:24.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d5359e700 1 --2- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540c8ed0 0x7f9d541574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:24.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d51d9b700 1 -- 172.21.15.92:0/1219014898 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021794817 0 0) 0x7f9d540c9350 con 0x7f9d54158710 2023-06-23T01:31:24.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d51d9b700 1 -- 172.21.15.92:0/1219014898 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d3c003570 con 0x7f9d54158710 2023-06-23T01:31:24.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d52d9d700 1 -- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540c8ed0 msgr2=0x7f9d541574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:24.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d52d9d700 1 --2- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540c8ed0 0x7f9d541574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:24.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d52d9d700 1 -- 172.21.15.92:0/1219014898 >> v1:172.21.15.92:6789/0 conn(0x7f9d54158710 legacy=0x7f9d54156d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:24.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d52d9d700 1 -- 172.21.15.92:0/1219014898 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5415b410 con 0x7f9d540cafa0 2023-06-23T01:31:24.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d5359e700 1 --2- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540c8ed0 0x7f9d541574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:24.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d52d9d700 1 --2- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d540cafa0 0x7f9d540c8990 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9d480098d0 tx=0x7f9d48009120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf74a48ca7ee168 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:24.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d51d9b700 1 -- 172.21.15.92:0/1219014898 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d48005c10 con 0x7f9d540cafa0 2023-06-23T01:31:24.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d51d9b700 1 -- 172.21.15.92:0/1219014898 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d48005d70 con 0x7f9d540cafa0 2023-06-23T01:31:24.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.551+0000 7f9d51d9b700 1 -- 172.21.15.92:0/1219014898 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d48006280 con 0x7f9d540cafa0 2023-06-23T01:31:24.558 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.552+0000 7f9d590b4700 1 -- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d540cafa0 msgr2=0x7f9d540c8990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:24.558 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.552+0000 7f9d590b4700 1 --2- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d540cafa0 0x7f9d540c8990 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9d480098d0 tx=0x7f9d48009120 comp rx=0 tx=0).stop 2023-06-23T01:31:24.558 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.552+0000 7f9d590b4700 1 -- 172.21.15.92:0/1219014898 shutdown_connections 2023-06-23T01:31:24.558 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.552+0000 7f9d590b4700 1 --2- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d540cafa0 0x7f9d540c8990 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:24.558 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.552+0000 7f9d590b4700 1 --2- 172.21.15.92:0/1219014898 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540c8ed0 0x7f9d541574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:24.558 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.552+0000 7f9d590b4700 1 -- 172.21.15.92:0/1219014898 >> 172.21.15.92:0/1219014898 conn(0x7f9d5406b530 msgr2=0x7f9d5406b930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:24.558 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.552+0000 7f9d590b4700 1 -- 172.21.15.92:0/1219014898 shutdown_connections 2023-06-23T01:31:24.559 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.552+0000 7f9d590b4700 1 -- 172.21.15.92:0/1219014898 wait complete. 2023-06-23T01:31:24.562 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.553+0000 7f9d590b4700 1 Processor -- start 2023-06-23T01:31:24.562 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.553+0000 7f9d590b4700 1 -- start start 2023-06-23T01:31:24.562 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d590b4700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540cafa0 0x7f9d54149fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:24.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d590b4700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d54158710 0x7f9d540a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:24.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d590b4700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54155590 0x7f9d540cd690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:24.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d590b4700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d54168760 con 0x7f9d54155590 2023-06-23T01:31:24.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d590b4700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d540c6530 con 0x7f9d54158710 2023-06-23T01:31:24.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d590b4700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d540c6780 con 0x7f9d540cafa0 2023-06-23T01:31:24.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d5359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54155590 0x7f9d540cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:24.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d5359e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54155590 0x7f9d540cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:41192/0 (socket says 172.21.15.92:41192) 2023-06-23T01:31:24.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d5359e700 1 -- 172.21.15.92:0/2689991541 learned_addr learned my addr 172.21.15.92:0/2689991541 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:24.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d5259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d54158710 0x7f9d540a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:24.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.554+0000 7f9d52d9d700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540cafa0 0x7f9d54149fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:24.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d5359e700 1 -- 172.21.15.92:0/2689991541 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540cafa0 msgr2=0x7f9d54149fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:24.566 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d5359e700 1 --2- 172.21.15.92:0/2689991541 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540cafa0 0x7f9d54149fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:24.566 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d5359e700 1 -- 172.21.15.92:0/2689991541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d54158710 msgr2=0x7f9d540a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:24.566 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d5359e700 1 --2- 172.21.15.92:0/2689991541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d54158710 0x7f9d540a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:24.567 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d5359e700 1 -- 172.21.15.92:0/2689991541 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d4c009a20 con 0x7f9d54155590 2023-06-23T01:31:24.567 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d5359e700 1 --2- 172.21.15.92:0/2689991541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54155590 0x7f9d540cd690 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009fc0 tx=0x7f9d4c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:24.567 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d3affd700 1 -- 172.21.15.92:0/2689991541 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c00cbd0 con 0x7f9d54155590 2023-06-23T01:31:24.567 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d3affd700 1 -- 172.21.15.92:0/2689991541 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d4c00f040 con 0x7f9d54155590 2023-06-23T01:31:24.568 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d480093c0 con 0x7f9d54155590 2023-06-23T01:31:24.568 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d3affd700 1 -- 172.21.15.92:0/2689991541 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c014480 con 0x7f9d54155590 2023-06-23T01:31:24.568 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d540c6d40 con 0x7f9d54155590 2023-06-23T01:31:24.568 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d3affd700 1 -- 172.21.15.92:0/2689991541 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9d4c003020 con 0x7f9d54155590 2023-06-23T01:31:24.569 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.555+0000 7f9d3affd700 1 -- 172.21.15.92:0/2689991541 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1641+0+0 (secure 0 0 0) 0x7f9d4c013ac0 con 0x7f9d54155590 2023-06-23T01:31:24.569 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.556+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5404c1b0 con 0x7f9d54155590 2023-06-23T01:31:24.569 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.558+0000 7f9d3affd700 1 -- 172.21.15.92:0/2689991541 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71365 (secure 0 0 0) 0x7f9d4c018030 con 0x7f9d54155590 2023-06-23T01:31:24.605 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:24.604+0000 7fcd5122f180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-06-23T01:31:24.629 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.627+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d5415b6f0 con 0x7f9d54155590 2023-06-23T01:31:24.630 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.628+0000 7f9d3affd700 1 -- 172.21.15.92:0/2689991541 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f9d4c013e20 con 0x7f9d54155590 2023-06-23T01:31:24.631 INFO:teuthology.misc.health.smithi092.stdout: 2023-06-23T01:31:24.631 INFO:teuthology.misc.health.smithi092.stdout:{"epoch":10,"fsid":"c94811cf-f7e5-4c57-b505-a983532f8da8","created":"2023-06-23T01:31:20.907484+0000","modified":"2023-06-23T01:31:23.354162+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:31:23.354162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4bcf4cdc-7c98-4a4a-bc56-133f22f4d802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a2e05a19-0c70-4cb4-aacd-3381219435b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"20f85617-68ff-491f-a436-aa1ad1979595","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3e5b05f4-6b77-4612-9720-38ece0b1388b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d94d9c9-4288-477f-801f-c66683295335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6b2f59-8af4-42b2-b7f9-bf36714c358f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2ac0eb61-39bf-4666-8c24-279bf8b9c51d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c4ff77b0-23dc-477d-adaa-d7fd9108f5da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:31:24.632 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54155590 msgr2=0x7f9d540cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:24.632 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 --2- 172.21.15.92:0/2689991541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54155590 0x7f9d540cd690 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009fc0 tx=0x7f9d4c00b040 comp rx=0 tx=0).stop 2023-06-23T01:31:24.632 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 shutdown_connections 2023-06-23T01:31:24.632 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 --2- 172.21.15.92:0/2689991541 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9d54155590 0x7f9d540cd690 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:24.633 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 --2- 172.21.15.92:0/2689991541 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9d540cafa0 0x7f9d54149fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:24.633 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 --2- 172.21.15.92:0/2689991541 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9d54158710 0x7f9d540a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:24.633 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 >> 172.21.15.92:0/2689991541 conn(0x7f9d5406b530 msgr2=0x7f9d54156060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:24.633 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 shutdown_connections 2023-06-23T01:31:24.633 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:24.629+0000 7f9d590b4700 1 -- 172.21.15.92:0/2689991541 wait complete. 2023-06-23T01:31:24.641 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-23T01:31:24.713 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:24.711+0000 7f39c8cc9180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:31:24.790 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:24.788+0000 7f39c8cc9180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:31:24.851 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:24.850+0000 7fcd5122f180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-06-23T01:31:24.928 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:24.927+0000 7fcd5122f180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-06-23T01:31:25.608 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:25.606+0000 7f39c8cc9180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:31:25.684 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:25.681+0000 7f39c8cc9180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:31:25.757 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:25.755+0000 7f39c8cc9180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:31:25.758 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:25.757+0000 7fcd5122f180 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-06-23T01:31:25.833 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:25.832+0000 7fcd5122f180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-06-23T01:31:25.892 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:25.890+0000 7f39c8cc9180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:31:25.908 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:25.907+0000 7fcd5122f180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-06-23T01:31:25.978 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:25.976+0000 7f39c8cc9180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:31:26.042 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:26.041+0000 7fcd5122f180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-06-23T01:31:26.127 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:26.125+0000 7f39c8cc9180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:31:26.142 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:26.141+0000 7fcd5122f180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-06-23T01:31:26.204 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:26.202+0000 7f39c8cc9180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:31:26.278 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:26.276+0000 7f39c8cc9180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:31:26.295 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:26.294+0000 7fcd5122f180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-06-23T01:31:26.353 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:26.351+0000 7f39c8cc9180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:31:26.371 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:26.369+0000 7fcd5122f180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-06-23T01:31:26.450 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:26.449+0000 7fcd5122f180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-06-23T01:31:26.528 INFO:tasks.ceph.mgr.x.smithi119.stderr:2023-06-23T01:31:26.527+0000 7fcd5122f180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-06-23T01:31:27.377 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:27.374+0000 7f39820ca700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-06-23T01:31:27.377 INFO:tasks.ceph.mgr.y.smithi092.stderr:2023-06-23T01:31:27.375+0000 7f39820ca700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-06-23T01:31:27.468 INFO:tasks.ceph.osd.2.smithi092.stderr:2023-06-23T01:31:27.466+0000 7fc09a699540 -1 osd.2 0 log_to_monitors true 2023-06-23T01:31:27.473 INFO:tasks.ceph.osd.3.smithi092.stderr:2023-06-23T01:31:27.471+0000 7ff9622b8540 -1 osd.3 0 log_to_monitors true 2023-06-23T01:31:27.475 INFO:tasks.ceph.osd.4.smithi119.stderr:2023-06-23T01:31:27.474+0000 7f5b094c3540 -1 osd.4 0 log_to_monitors true 2023-06-23T01:31:27.476 INFO:tasks.ceph.osd.5.smithi119.stderr:2023-06-23T01:31:27.474+0000 7f6a9e10d540 -1 osd.5 0 log_to_monitors true 2023-06-23T01:31:27.498 INFO:tasks.ceph.osd.0.smithi092.stderr:2023-06-23T01:31:27.496+0000 7f0ec6f3b540 -1 osd.0 0 log_to_monitors true 2023-06-23T01:31:27.525 INFO:tasks.ceph.osd.1.smithi092.stderr:2023-06-23T01:31:27.523+0000 7f6beab55540 -1 osd.1 0 log_to_monitors true 2023-06-23T01:31:27.526 INFO:tasks.ceph.osd.6.smithi119.stderr:2023-06-23T01:31:27.523+0000 7ff6bc7aa540 -1 osd.6 0 log_to_monitors true 2023-06-23T01:31:27.545 INFO:tasks.ceph.osd.7.smithi119.stderr:2023-06-23T01:31:27.543+0000 7fb2b31f5540 -1 osd.7 0 log_to_monitors true 2023-06-23T01:31:31.354 INFO:tasks.ceph.osd.1.smithi092.stderr:2023-06-23T01:31:31.352+0000 7f6be1777700 -1 osd.1 0 waiting for initial osdmap 2023-06-23T01:31:31.359 INFO:tasks.ceph.osd.1.smithi092.stderr:2023-06-23T01:31:31.357+0000 7f6bda56e700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:31:31.394 INFO:tasks.ceph.osd.2.smithi092.stderr:2023-06-23T01:31:31.392+0000 7fc0912bb700 -1 osd.2 0 waiting for initial osdmap 2023-06-23T01:31:31.397 INFO:tasks.ceph.osd.2.smithi092.stderr:2023-06-23T01:31:31.395+0000 7fc08a0b2700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:31:31.400 INFO:tasks.ceph.osd.7.smithi119.stderr:2023-06-23T01:31:31.397+0000 7fb2a9e17700 -1 osd.7 0 waiting for initial osdmap 2023-06-23T01:31:31.405 INFO:tasks.ceph.osd.7.smithi119.stderr:2023-06-23T01:31:31.403+0000 7fb2a2c0e700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:31:31.417 INFO:tasks.ceph.osd.5.smithi119.stderr:2023-06-23T01:31:31.415+0000 7f6a94d2f700 -1 osd.5 0 waiting for initial osdmap 2023-06-23T01:31:31.422 INFO:tasks.ceph.osd.5.smithi119.stderr:2023-06-23T01:31:31.420+0000 7f6a8db26700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:31:31.437 INFO:tasks.ceph.osd.0.smithi092.stderr:2023-06-23T01:31:31.434+0000 7f0ebdb5d700 -1 osd.0 0 waiting for initial osdmap 2023-06-23T01:31:31.441 INFO:tasks.ceph.osd.0.smithi092.stderr:2023-06-23T01:31:31.439+0000 7f0eb6954700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:31:31.447 DEBUG:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:31:31.448 INFO:tasks.ceph.osd.3.smithi092.stderr:2023-06-23T01:31:31.446+0000 7ff958eda700 -1 osd.3 0 waiting for initial osdmap 2023-06-23T01:31:31.451 INFO:tasks.ceph.osd.3.smithi092.stderr:2023-06-23T01:31:31.449+0000 7ff951cd1700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:31:31.459 INFO:tasks.ceph.osd.4.smithi119.stderr:2023-06-23T01:31:31.457+0000 7f5b000e5700 -1 osd.4 0 waiting for initial osdmap 2023-06-23T01:31:31.462 INFO:tasks.ceph.osd.4.smithi119.stderr:2023-06-23T01:31:31.460+0000 7f5af8edc700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:31:31.470 INFO:tasks.ceph.osd.6.smithi119.stderr:2023-06-23T01:31:31.468+0000 7ff6b33cc700 -1 osd.6 0 waiting for initial osdmap 2023-06-23T01:31:31.473 INFO:tasks.ceph.osd.6.smithi119.stderr:2023-06-23T01:31:31.472+0000 7ff6ac1c3700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-06-23T01:31:31.552 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.550+0000 7f8d97d34700 1 Processor -- start 2023-06-23T01:31:31.553 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.550+0000 7f8d97d34700 1 -- start start 2023-06-23T01:31:31.554 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.551+0000 7f8d97d34700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f8d900b1e90 0x7f8d900b0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:31.554 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.551+0000 7f8d97d34700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d900cd160 0x7f8d900cd5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:31.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.551+0000 7f8d97d34700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d90158d80 con 0x7f8d90157c00 2023-06-23T01:31:31.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.551+0000 7f8d97d34700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d900cdb00 con 0x7f8d900b1e90 2023-06-23T01:31:31.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.551+0000 7f8d97d34700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90158170 con 0x7f8d900cd160 2023-06-23T01:31:31.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.551+0000 7f8d952cf700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f8d90157c00 0x7f8d9006eed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:40996/0 (socket says 172.21.15.92:40996) 2023-06-23T01:31:31.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.551+0000 7f8d952cf700 1 -- 172.21.15.92:0/2763989565 learned_addr learned my addr 172.21.15.92:0/2763989565 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:31.555 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d962d1700 1 --2- 172.21.15.92:0/2763989565 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d900cd160 0x7f8d900cd5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:31.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d95ad0700 1 --2- 172.21.15.92:0/2763989565 >> v2:172.21.15.119:3300/0 conn(0x7f8d900b1e90 0x7f8d900b0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:31.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d94ace700 1 -- 172.21.15.92:0/2763989565 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913020268 0 0) 0x7f8d90158d80 con 0x7f8d90157c00 2023-06-23T01:31:31.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d94ace700 1 -- 172.21.15.92:0/2763989565 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d7c003570 con 0x7f8d90157c00 2023-06-23T01:31:31.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d962d1700 1 -- 172.21.15.92:0/2763989565 >> v2:172.21.15.119:3300/0 conn(0x7f8d900b1e90 msgr2=0x7f8d900b0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:31.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d962d1700 1 --2- 172.21.15.92:0/2763989565 >> v2:172.21.15.119:3300/0 conn(0x7f8d900b1e90 0x7f8d900b0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d962d1700 1 -- 172.21.15.92:0/2763989565 >> v1:172.21.15.92:6789/0 conn(0x7f8d90157c00 legacy=0x7f8d9006eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:31.556 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d962d1700 1 -- 172.21.15.92:0/2763989565 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d90158450 con 0x7f8d900cd160 2023-06-23T01:31:31.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d95ad0700 1 --2- 172.21.15.92:0/2763989565 >> v2:172.21.15.119:3300/0 conn(0x7f8d900b1e90 0x7f8d900b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:31.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d962d1700 1 --2- 172.21.15.92:0/2763989565 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d900cd160 0x7f8d900cd5c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009e60 tx=0x7f8d88009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6d3574d1ec6ffb1 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:31.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d94ace700 1 -- 172.21.15.92:0/2763989565 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800e070 con 0x7f8d900cd160 2023-06-23T01:31:31.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d94ace700 1 -- 172.21.15.92:0/2763989565 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8800bab0 con 0x7f8d900cd160 2023-06-23T01:31:31.557 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.552+0000 7f8d94ace700 1 -- 172.21.15.92:0/2763989565 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8800be70 con 0x7f8d900cd160 2023-06-23T01:31:31.559 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.553+0000 7f8d97d34700 1 -- 172.21.15.92:0/2763989565 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d900cd160 msgr2=0x7f8d900cd5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:31.560 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.553+0000 7f8d97d34700 1 --2- 172.21.15.92:0/2763989565 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d900cd160 0x7f8d900cd5c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009e60 tx=0x7f8d88009780 comp rx=0 tx=0).stop 2023-06-23T01:31:31.560 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.553+0000 7f8d97d34700 1 -- 172.21.15.92:0/2763989565 shutdown_connections 2023-06-23T01:31:31.560 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.553+0000 7f8d97d34700 1 --2- 172.21.15.92:0/2763989565 >> v2:172.21.15.119:3300/0 conn(0x7f8d900b1e90 0x7f8d900b0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.560 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.553+0000 7f8d97d34700 1 --2- 172.21.15.92:0/2763989565 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d900cd160 0x7f8d900cd5c0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.560 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.553+0000 7f8d97d34700 1 -- 172.21.15.92:0/2763989565 >> 172.21.15.92:0/2763989565 conn(0x7f8d90069740 msgr2=0x7f8d9006a560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:31.560 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.553+0000 7f8d97d34700 1 -- 172.21.15.92:0/2763989565 shutdown_connections 2023-06-23T01:31:31.561 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.553+0000 7f8d97d34700 1 -- 172.21.15.92:0/2763989565 wait complete. 2023-06-23T01:31:31.561 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.554+0000 7f8d97d34700 1 Processor -- start 2023-06-23T01:31:31.561 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.554+0000 7f8d97d34700 1 -- start start 2023-06-23T01:31:31.561 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d97d34700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d900b1e90 0x7f8d90154790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:31.561 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d97d34700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d900cd160 0x7f8d90154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:31.561 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d97d34700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d90157c00 0x7f8d900a8010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:31.561 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d97d34700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90167500 con 0x7f8d900b1e90 2023-06-23T01:31:31.562 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d97d34700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d900aa000 con 0x7f8d900cd160 2023-06-23T01:31:31.562 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d97d34700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d90167220 con 0x7f8d90157c00 2023-06-23T01:31:31.562 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d900b1e90 0x7f8d90154790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:31.562 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d900b1e90 0x7f8d90154790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53876/0 (socket says 172.21.15.92:53876) 2023-06-23T01:31:31.562 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 -- 172.21.15.92:0/3304373395 learned_addr learned my addr 172.21.15.92:0/3304373395 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:31.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d962d1700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d90157c00 0x7f8d900a8010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:31.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d952cf700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d900cd160 0x7f8d90154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:31.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 -- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d90157c00 msgr2=0x7f8d900a8010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:31.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d90157c00 0x7f8d900a8010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 -- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d900cd160 msgr2=0x7f8d90154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:31.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d900cd160 0x7f8d90154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 -- 172.21.15.92:0/3304373395 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d8c009420 con 0x7f8d900b1e90 2023-06-23T01:31:31.563 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d962d1700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d90157c00 0x7f8d900a8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:31:31.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d95ad0700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d900b1e90 0x7f8d90154790 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00ec80 tx=0x7f8d8c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:31.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d85ffb700 1 -- 172.21.15.92:0/3304373395 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00c9d0 con 0x7f8d900b1e90 2023-06-23T01:31:31.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d85ffb700 1 -- 172.21.15.92:0/3304373395 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d8c00cb30 con 0x7f8d900b1e90 2023-06-23T01:31:31.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d88009a20 con 0x7f8d900b1e90 2023-06-23T01:31:31.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.555+0000 7f8d85ffb700 1 -- 172.21.15.92:0/3304373395 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00ceb0 con 0x7f8d900b1e90 2023-06-23T01:31:31.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.556+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d900a5a70 con 0x7f8d900b1e90 2023-06-23T01:31:31.564 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.556+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d900a55f0 con 0x7f8d900b1e90 2023-06-23T01:31:31.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.557+0000 7f8d85ffb700 1 -- 172.21.15.92:0/3304373395 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 4) v1 ==== 84592+0+0 (secure 0 0 0) 0x7f8d8c013680 con 0x7f8d900b1e90 2023-06-23T01:31:31.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.557+0000 7f8d85ffb700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f8d74066e00 0x7f8d740692b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:31.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.557+0000 7f8d85ffb700 1 -- 172.21.15.92:0/3304373395 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2227+0+0 (secure 0 0 0) 0x7f8d8c0839a0 con 0x7f8d900b1e90 2023-06-23T01:31:31.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.557+0000 7f8d952cf700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f8d74066e00 0x7f8d740692b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:31.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.558+0000 7f8d952cf700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f8d74066e00 0x7f8d740692b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d900b1770 tx=0x7f8d80009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:31.565 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.561+0000 7f8d85ffb700 1 -- 172.21.15.92:0/3304373395 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f8d900a55f0 con 0x7f8d900b1e90 2023-06-23T01:31:31.748 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.746+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d900a55f0 con 0x7f8d900b1e90 2023-06-23T01:31:31.748 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.746+0000 7f8d85ffb700 1 -- 172.21.15.92:0/3304373395 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7013 (secure 0 0 0) 0x7f8d900a55f0 con 0x7f8d900b1e90 2023-06-23T01:31:31.749 INFO:teuthology.misc.health.smithi092.stdout: 2023-06-23T01:31:31.749 INFO:teuthology.misc.health.smithi092.stdout:{"epoch":12,"fsid":"c94811cf-f7e5-4c57-b505-a983532f8da8","created":"2023-06-23T01:31:20.907484+0000","modified":"2023-06-23T01:31:29.372204+0000","last_up_change":"0.000000","last_in_change":"2023-06-23T01:31:23.354162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"4bcf4cdc-7c98-4a4a-bc56-133f22f4d802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"a2e05a19-0c70-4cb4-aacd-3381219435b8","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"20f85617-68ff-491f-a436-aa1ad1979595","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"3e5b05f4-6b77-4612-9720-38ece0b1388b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"4d94d9c9-4288-477f-801f-c66683295335","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"dc6b2f59-8af4-42b2-b7f9-bf36714c358f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"2ac0eb61-39bf-4666-8c24-279bf8b9c51d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"c4ff77b0-23dc-477d-adaa-d7fd9108f5da","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:31:31.750 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f8d74066e00 msgr2=0x7f8d740692b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:31.751 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f8d74066e00 0x7f8d740692b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d900b1770 tx=0x7f8d80009040 comp rx=0 tx=0).stop 2023-06-23T01:31:31.751 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d900b1e90 msgr2=0x7f8d90154790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:31.751 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d900b1e90 0x7f8d90154790 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00ec80 tx=0x7f8d8c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.751 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 shutdown_connections 2023-06-23T01:31:31.751 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f8d74066e00 0x7f8d740692b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.751 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f8d90157c00 0x7f8d900a8010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.752 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f8d900b1e90 0x7f8d90154790 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.752 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 --2- 172.21.15.92:0/3304373395 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f8d900cd160 0x7f8d90154cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:31.752 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 >> 172.21.15.92:0/3304373395 conn(0x7f8d90069740 msgr2=0x7f8d900cc450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:31.752 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 shutdown_connections 2023-06-23T01:31:31.752 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:31.748+0000 7f8d97d34700 1 -- 172.21.15.92:0/3304373395 wait complete. 2023-06-23T01:31:31.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-06-23T01:31:38.569 DEBUG:teuthology.orchestra.run.smithi092:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-06-23T01:31:38.681 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.679+0000 7f518f836700 1 Processor -- start 2023-06-23T01:31:38.681 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.680+0000 7f518f836700 1 -- start start 2023-06-23T01:31:38.684 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.680+0000 7f518f836700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5188159b30 0x7f5188158c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:38.684 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.680+0000 7f518f836700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188157490 con 0x7f51881502b0 2023-06-23T01:31:38.684 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.680+0000 7f518f836700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5188157770 con 0x7f51881596b0 2023-06-23T01:31:38.685 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.680+0000 7f518f836700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5188157a50 con 0x7f5188159b30 2023-06-23T01:31:38.685 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f518d5d2700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f51881502b0 0x7f51880aff60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:41002/0 (socket says 172.21.15.92:41002) 2023-06-23T01:31:38.685 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f518d5d2700 1 -- 172.21.15.92:0/3055427516 learned_addr learned my addr 172.21.15.92:0/3055427516 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:38.685 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f518ddd3700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5188159b30 0x7f5188158c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:38.685 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922066049 0 0) 0x7f5188157490 con 0x7f51881502b0 2023-06-23T01:31:38.685 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5174003490 con 0x7f51881502b0 2023-06-23T01:31:38.686 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1488413431 0 0) 0x7f5174003490 con 0x7f51881502b0 2023-06-23T01:31:38.686 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5188159b30 msgr2=0x7f5188158c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:38.686 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5188159b30 0x7f5188158c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.686 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 >> v1:172.21.15.119:6789/0 conn(0x7f51881596b0 legacy=0x7f5188158530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:38.686 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5188157d30 con 0x7f51881502b0 2023-06-23T01:31:38.686 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 <== mon.0 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3236472324 0 0) 0x7f5184001470 con 0x7f51881502b0 2023-06-23T01:31:38.686 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 >> v1:172.21.15.92:6789/0 conn(0x7f51881502b0 legacy=0x7f51880aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:38.686 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5174003c10 0x7f5174004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:38.687 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f517400adc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:38.687 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5188159b30 0x7f517400d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:38.687 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5188157a50 con 0x7f5174008940 2023-06-23T01:31:38.687 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517400dd70 con 0x7f5174003c10 2023-06-23T01:31:38.687 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f517400df30 con 0x7f5188159b30 2023-06-23T01:31:38.687 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f518cdd1700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f517400adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:38.687 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f518d5d2700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5174003c10 0x7f5174004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:38.688 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.681+0000 7f518ddd3700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5188159b30 0x7f517400d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:38.688 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518cdd1700 1 -- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5188159b30 msgr2=0x7f517400d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:38.688 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518cdd1700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5188159b30 0x7f517400d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.688 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518cdd1700 1 -- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5174003c10 msgr2=0x7f5174004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:38.688 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518cdd1700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5174003c10 0x7f5174004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.688 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518cdd1700 1 -- 172.21.15.92:0/3055427516 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5174003490 con 0x7f5174008940 2023-06-23T01:31:38.688 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518cdd1700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f517400adc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f517c001390 tx=0x7f517c007d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e62d0108108d228d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:38.689 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c0116c0 con 0x7f5174008940 2023-06-23T01:31:38.689 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517c011820 con 0x7f5174008940 2023-06-23T01:31:38.689 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f517bfff700 1 -- 172.21.15.92:0/3055427516 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f517c011bc0 con 0x7f5174008940 2023-06-23T01:31:38.689 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518f836700 1 -- 172.21.15.92:0/3055427516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 msgr2=0x7f517400adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:38.689 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518f836700 1 --2- 172.21.15.92:0/3055427516 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f517400adc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f517c001390 tx=0x7f517c007d30 comp rx=0 tx=0).stop 2023-06-23T01:31:38.689 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518ddd3700 1 -- 172.21.15.92:0/3055427516 reap_dead start 2023-06-23T01:31:38.689 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518f836700 1 -- 172.21.15.92:0/3055427516 shutdown_connections 2023-06-23T01:31:38.690 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.682+0000 7f518f836700 1 -- 172.21.15.92:0/3055427516 >> 172.21.15.92:0/3055427516 conn(0x7f5188067b90 msgr2=0x7f5188067f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:38.690 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.683+0000 7f518f836700 1 -- 172.21.15.92:0/3055427516 shutdown_connections 2023-06-23T01:31:38.690 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.683+0000 7f518f836700 1 -- 172.21.15.92:0/3055427516 wait complete. 2023-06-23T01:31:38.692 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.684+0000 7f518f836700 1 Processor -- start 2023-06-23T01:31:38.692 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.684+0000 7f518f836700 1 -- start start 2023-06-23T01:31:38.692 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.684+0000 7f518f836700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f518814a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:38.692 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518f836700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f518814a770 0x7f518814bea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:38.693 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518f836700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f518814c3e0 0x7f518814c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:38.693 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518f836700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518814f2a0 con 0x7f5174008940 2023-06-23T01:31:38.693 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518f836700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518814cda0 con 0x7f518814a770 2023-06-23T01:31:38.693 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518f836700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f518814e690 con 0x7f518814c3e0 2023-06-23T01:31:38.693 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f518814a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:38.693 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f518814a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53936/0 (socket says 172.21.15.92:53936) 2023-06-23T01:31:38.693 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 -- 172.21.15.92:0/294458896 learned_addr learned my addr 172.21.15.92:0/294458896 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:38.694 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518cdd1700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f518814a770 0x7f518814bea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:38.694 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518ddd3700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f518814c3e0 0x7f518814c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:38.694 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 -- 172.21.15.92:0/294458896 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f518814c3e0 msgr2=0x7f518814c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:38.694 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f518814c3e0 0x7f518814c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.694 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 -- 172.21.15.92:0/294458896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f518814a770 msgr2=0x7f518814bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:38.694 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f518814a770 0x7f518814bea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.694 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 -- 172.21.15.92:0/294458896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518814e970 con 0x7f5174008940 2023-06-23T01:31:38.694 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518d5d2700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f518814a230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5184007310 tx=0x7f5184001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:38.695 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f51797fa700 1 -- 172.21.15.92:0/294458896 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518400f710 con 0x7f5174008940 2023-06-23T01:31:38.695 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f51797fa700 1 -- 172.21.15.92:0/294458896 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518400f870 con 0x7f5174008940 2023-06-23T01:31:38.695 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518814ec50 con 0x7f5174008940 2023-06-23T01:31:38.695 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f51797fa700 1 -- 172.21.15.92:0/294458896 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518400fc30 con 0x7f5174008940 2023-06-23T01:31:38.695 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.685+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5188147f70 con 0x7f5174008940 2023-06-23T01:31:38.695 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.686+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518804c1b0 con 0x7f5174008940 2023-06-23T01:31:38.695 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.687+0000 7f51797fa700 1 -- 172.21.15.92:0/294458896 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f51840133b0 con 0x7f5174008940 2023-06-23T01:31:38.696 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.687+0000 7f51797fa700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f516c066e90 0x7f516c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:38.696 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.687+0000 7f51797fa700 1 -- 172.21.15.92:0/294458896 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f5184083ce0 con 0x7f5174008940 2023-06-23T01:31:38.696 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.687+0000 7f518cdd1700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f516c066e90 0x7f516c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:38.696 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.688+0000 7f518cdd1700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f516c066e90 0x7f516c069340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f518806d800 tx=0x7f517c003000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:38.696 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.691+0000 7f51797fa700 1 -- 172.21.15.92:0/294458896 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f5184051340 con 0x7f5174008940 2023-06-23T01:31:38.877 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.876+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5188158010 con 0x7f5174008940 2023-06-23T01:31:38.878 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.877+0000 7f51797fa700 1 -- 172.21.15.92:0/294458896 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12127 (secure 0 0 0) 0x7f518401a090 con 0x7f5174008940 2023-06-23T01:31:38.879 INFO:teuthology.misc.health.smithi092.stdout: 2023-06-23T01:31:38.879 INFO:teuthology.misc.health.smithi092.stdout:{"epoch":16,"fsid":"c94811cf-f7e5-4c57-b505-a983532f8da8","created":"2023-06-23T01:31:20.907484+0000","modified":"2023-06-23T01:31:35.400682+0000","last_up_change":"2023-06-23T01:31:32.355093+0000","last_in_change":"2023-06-23T01:31:23.354162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:31:32.414191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"4bcf4cdc-7c98-4a4a-bc56-133f22f4d802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":87954},{"type":"v1","addr":"172.21.15.92:6809","nonce":87954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":87954},{"type":"v1","addr":"172.21.15.92:6811","nonce":87954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":87954},{"type":"v1","addr":"172.21.15.92:6815","nonce":87954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":87954},{"type":"v1","addr":"172.21.15.92:6813","nonce":87954}]},"public_addr":"172.21.15.92:6809/87954","cluster_addr":"172.21.15.92:6811/87954","heartbeat_back_addr":"172.21.15.92:6815/87954","heartbeat_front_addr":"172.21.15.92:6813/87954","state":["exists","up"]},{"osd":1,"uuid":"a2e05a19-0c70-4cb4-aacd-3381219435b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":87955},{"type":"v1","addr":"172.21.15.92:6821","nonce":87955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":87955},{"type":"v1","addr":"172.21.15.92:6826","nonce":87955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":87955},{"type":"v1","addr":"172.21.15.92:6831","nonce":87955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":87955},{"type":"v1","addr":"172.21.15.92:6829","nonce":87955}]},"public_addr":"172.21.15.92:6821/87955","cluster_addr":"172.21.15.92:6826/87955","heartbeat_back_addr":"172.21.15.92:6831/87955","heartbeat_front_addr":"172.21.15.92:6829/87955","state":["exists","up"]},{"osd":2,"uuid":"20f85617-68ff-491f-a436-aa1ad1979595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":87951},{"type":"v1","addr":"172.21.15.92:6801","nonce":87951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":87951},{"type":"v1","addr":"172.21.15.92:6803","nonce":87951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":87951},{"type":"v1","addr":"172.21.15.92:6807","nonce":87951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":87951},{"type":"v1","addr":"172.21.15.92:6805","nonce":87951}]},"public_addr":"172.21.15.92:6801/87951","cluster_addr":"172.21.15.92:6803/87951","heartbeat_back_addr":"172.21.15.92:6807/87951","heartbeat_front_addr":"172.21.15.92:6805/87951","state":["exists","up"]},{"osd":3,"uuid":"3e5b05f4-6b77-4612-9720-38ece0b1388b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":87956},{"type":"v1","addr":"172.21.15.92:6817","nonce":87956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":87956},{"type":"v1","addr":"172.21.15.92:6820","nonce":87956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":87956},{"type":"v1","addr":"172.21.15.92:6827","nonce":87956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":87956},{"type":"v1","addr":"172.21.15.92:6824","nonce":87956}]},"public_addr":"172.21.15.92:6817/87956","cluster_addr":"172.21.15.92:6820/87956","heartbeat_back_addr":"172.21.15.92:6827/87956","heartbeat_front_addr":"172.21.15.92:6824/87956","state":["exists","up"]},{"osd":4,"uuid":"4d94d9c9-4288-477f-801f-c66683295335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6801","nonce":88172},{"type":"v1","addr":"172.21.15.119:6802","nonce":88172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":88172},{"type":"v1","addr":"172.21.15.119:6806","nonce":88172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":88172},{"type":"v1","addr":"172.21.15.119:6814","nonce":88172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":88172},{"type":"v1","addr":"172.21.15.119:6810","nonce":88172}]},"public_addr":"172.21.15.119:6802/88172","cluster_addr":"172.21.15.119:6806/88172","heartbeat_back_addr":"172.21.15.119:6814/88172","heartbeat_front_addr":"172.21.15.119:6810/88172","state":["exists","up"]},{"osd":5,"uuid":"dc6b2f59-8af4-42b2-b7f9-bf36714c358f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":88173},{"type":"v1","addr":"172.21.15.119:6803","nonce":88173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6805","nonce":88173},{"type":"v1","addr":"172.21.15.119:6807","nonce":88173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":88173},{"type":"v1","addr":"172.21.15.119:6815","nonce":88173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":88173},{"type":"v1","addr":"172.21.15.119:6811","nonce":88173}]},"public_addr":"172.21.15.119:6803/88173","cluster_addr":"172.21.15.119:6807/88173","heartbeat_back_addr":"172.21.15.119:6815/88173","heartbeat_front_addr":"172.21.15.119:6811/88173","state":["exists","up"]},{"osd":6,"uuid":"2ac0eb61-39bf-4666-8c24-279bf8b9c51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6817","nonce":88202},{"type":"v1","addr":"172.21.15.119:6819","nonce":88202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6821","nonce":88202},{"type":"v1","addr":"172.21.15.119:6823","nonce":88202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6829","nonce":88202},{"type":"v1","addr":"172.21.15.119:6831","nonce":88202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6825","nonce":88202},{"type":"v1","addr":"172.21.15.119:6827","nonce":88202}]},"public_addr":"172.21.15.119:6819/88202","cluster_addr":"172.21.15.119:6823/88202","heartbeat_back_addr":"172.21.15.119:6831/88202","heartbeat_front_addr":"172.21.15.119:6827/88202","state":["exists","up"]},{"osd":7,"uuid":"c4ff77b0-23dc-477d-adaa-d7fd9108f5da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":88203},{"type":"v1","addr":"172.21.15.119:6818","nonce":88203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":88203},{"type":"v1","addr":"172.21.15.119:6822","nonce":88203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":88203},{"type":"v1","addr":"172.21.15.119:6830","nonce":88203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":88203},{"type":"v1","addr":"172.21.15.119:6826","nonce":88203}]},"public_addr":"172.21.15.119:6818/88203","cluster_addr":"172.21.15.119:6822/88203","heartbeat_back_addr":"172.21.15.119:6830/88203","heartbeat_front_addr":"172.21.15.119:6826/88203","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.542554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.495262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.517893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.443537+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.444113+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.472890+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.556249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.573872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:31:38.881 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f516c066e90 msgr2=0x7f516c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:38.881 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f516c066e90 0x7f516c069340 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f518806d800 tx=0x7f517c003000 comp rx=0 tx=0).stop 2023-06-23T01:31:38.881 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 msgr2=0x7f518814a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:38.882 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f518814a230 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f5184007310 tx=0x7f5184001e50 comp rx=0 tx=0).stop 2023-06-23T01:31:38.882 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 shutdown_connections 2023-06-23T01:31:38.882 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f516c066e90 0x7f516c069340 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.882 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f518814c3e0 0x7f518814c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.882 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5174008940 0x7f518814a230 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.882 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 --2- 172.21.15.92:0/294458896 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f518814a770 0x7f518814bea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:38.882 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 >> 172.21.15.92:0/294458896 conn(0x7f5188067b90 msgr2=0x7f5188149930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:38.883 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 shutdown_connections 2023-06-23T01:31:38.883 INFO:teuthology.misc.health.smithi092.stderr:2023-06-23T01:31:38.879+0000 7f518f836700 1 -- 172.21.15.92:0/294458896 wait complete. 2023-06-23T01:31:38.891 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-06-23T01:31:38.891 INFO:tasks.ceph:Creating RBD pool 2023-06-23T01:31:38.892 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd pool create rbd 8 2023-06-23T01:31:39.004 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.003+0000 7f5be759e700 1 Processor -- start 2023-06-23T01:31:39.004 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.003+0000 7f5be759e700 1 -- start start 2023-06-23T01:31:39.005 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be759e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5be8159310 0x7f5be81596e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.006 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be759e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be80b23d0 0x7f5be815dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.006 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be759e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5be80cb7d0 con 0x7f5be80b00e0 2023-06-23T01:31:39.006 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be759e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5be80b2e10 con 0x7f5be8159310 2023-06-23T01:31:39.006 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be759e700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80b15c0 con 0x7f5be80b23d0 2023-06-23T01:31:39.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be659c700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f5be8159310 0x7f5be81596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be5d9b700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be80b23d0 0x7f5be815dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be6d9d700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f5be80b00e0 0x7f5be80683a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.92:39636/0 (socket says 172.21.15.92:39636) 2023-06-23T01:31:39.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be6d9d700 1 -- 172.21.15.92:0/48851321 learned_addr learned my addr 172.21.15.92:0/48851321 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:39.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be5d9b700 1 -- 172.21.15.92:0/48851321 >> v2:172.21.15.119:3300/0 conn(0x7f5be8159310 msgr2=0x7f5be81596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:39.007 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be5d9b700 1 --2- 172.21.15.92:0/48851321 >> v2:172.21.15.119:3300/0 conn(0x7f5be8159310 0x7f5be81596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be5d9b700 1 -- 172.21.15.92:0/48851321 >> v1:172.21.15.119:6789/0 conn(0x7f5be80b00e0 legacy=0x7f5be80683a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be5d9b700 1 -- 172.21.15.92:0/48851321 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be815e150 con 0x7f5be80b23d0 2023-06-23T01:31:39.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.004+0000 7f5be659c700 1 --2- 172.21.15.92:0/48851321 >> v2:172.21.15.119:3300/0 conn(0x7f5be8159310 0x7f5be81596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:39.008 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be5d9b700 1 --2- 172.21.15.92:0/48851321 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be80b23d0 0x7f5be815dc10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0098e0 tx=0x7f5bdc0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57810bae54490c4c server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:39.010 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be559a700 1 -- 172.21.15.92:0/48851321 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc00b640 con 0x7f5be80b23d0 2023-06-23T01:31:39.010 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be559a700 1 -- 172.21.15.92:0/48851321 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bdc00b7a0 con 0x7f5be80b23d0 2023-06-23T01:31:39.011 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be559a700 1 -- 172.21.15.92:0/48851321 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bdc00bb40 con 0x7f5be80b23d0 2023-06-23T01:31:39.011 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be759e700 1 -- 172.21.15.92:0/48851321 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be80b23d0 msgr2=0x7f5be815dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.011 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be759e700 1 --2- 172.21.15.92:0/48851321 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be80b23d0 0x7f5be815dc10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0098e0 tx=0x7f5bdc0091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.011 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be759e700 1 -- 172.21.15.92:0/48851321 shutdown_connections 2023-06-23T01:31:39.011 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be759e700 1 --2- 172.21.15.92:0/48851321 >> v2:172.21.15.119:3300/0 conn(0x7f5be8159310 0x7f5be81596e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.012 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be759e700 1 --2- 172.21.15.92:0/48851321 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be80b23d0 0x7f5be815dc10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.012 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be759e700 1 -- 172.21.15.92:0/48851321 >> 172.21.15.92:0/48851321 conn(0x7f5be8070290 msgr2=0x7f5be8070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:39.012 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be759e700 1 -- 172.21.15.92:0/48851321 shutdown_connections 2023-06-23T01:31:39.012 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.005+0000 7f5be759e700 1 -- 172.21.15.92:0/48851321 wait complete. 2023-06-23T01:31:39.012 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.006+0000 7f5be759e700 1 Processor -- start 2023-06-23T01:31:39.012 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be759e700 1 -- start start 2023-06-23T01:31:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be759e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5be80b00e0 0x7f5be80c7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be759e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be8159310 0x7f5be815f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be759e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be80c9950 0x7f5be81484a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be759e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80c88b0 con 0x7f5be80b00e0 2023-06-23T01:31:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be759e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80c8b90 con 0x7f5be80c9950 2023-06-23T01:31:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be759e700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be80c8e70 con 0x7f5be8159310 2023-06-23T01:31:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5be80b00e0 0x7f5be80c7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.013 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be659c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5be80b00e0 0x7f5be80c7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53942/0 (socket says 172.21.15.92:53942) 2023-06-23T01:31:39.014 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be659c700 1 -- 172.21.15.92:0/3361635441 learned_addr learned my addr 172.21.15.92:0/3361635441 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:39.014 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be6d9d700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be80c9950 0x7f5be81484a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.014 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.007+0000 7f5be5d9b700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be8159310 0x7f5be815f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.014 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5be659c700 1 -- 172.21.15.92:0/3361635441 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be8159310 msgr2=0x7f5be815f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.014 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5be659c700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be8159310 0x7f5be815f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.014 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5be659c700 1 -- 172.21.15.92:0/3361635441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be80c9950 msgr2=0x7f5be81484a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.014 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5be659c700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be80c9950 0x7f5be81484a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.015 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5be659c700 1 -- 172.21.15.92:0/3361635441 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd8009420 con 0x7f5be80b00e0 2023-06-23T01:31:39.015 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5be659c700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5be80b00e0 0x7f5be80c7ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800ec80 tx=0x7f5bd80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:39.015 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5bd67fc700 1 -- 172.21.15.92:0/3361635441 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd800c850 con 0x7f5be80b00e0 2023-06-23T01:31:39.015 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5bd67fc700 1 -- 172.21.15.92:0/3361635441 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bd800c9b0 con 0x7f5be80b00e0 2023-06-23T01:31:39.015 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bdc009420 con 0x7f5be80b00e0 2023-06-23T01:31:39.015 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5bd67fc700 1 -- 172.21.15.92:0/3361635441 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bd800cd30 con 0x7f5be80b00e0 2023-06-23T01:31:39.015 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.008+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5be80c9430 con 0x7f5be80b00e0 2023-06-23T01:31:39.016 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.009+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5be80c97a0 con 0x7f5be80b00e0 2023-06-23T01:31:39.016 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.009+0000 7f5bd67fc700 1 -- 172.21.15.92:0/3361635441 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f5bd8013680 con 0x7f5be80b00e0 2023-06-23T01:31:39.016 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.009+0000 7f5bd67fc700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5bc8066e90 0x7f5bc8069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.017 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.010+0000 7f5bd67fc700 1 -- 172.21.15.92:0/3361635441 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f5bd8084630 con 0x7f5be80b00e0 2023-06-23T01:31:39.017 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.010+0000 7f5be5d9b700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5bc8066e90 0x7f5bc8069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.017 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.010+0000 7f5be5d9b700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5bc8066e90 0x7f5bc8069340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0093f0 tx=0x7f5bdc017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:39.017 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.014+0000 7f5bd67fc700 1 -- 172.21.15.92:0/3361635441 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f5bd8055740 con 0x7f5be80b00e0 2023-06-23T01:31:39.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.198+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f5be8067f00 con 0x7f5be80b00e0 2023-06-23T01:31:39.702 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.701+0000 7f5bd67fc700 1 -- 172.21.15.92:0/3361635441 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f5bd80552d0 con 0x7f5be80b00e0 2023-06-23T01:31:39.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5bc8066e90 msgr2=0x7f5bc8069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.705 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5bc8066e90 0x7f5bc8069340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5bdc0093f0 tx=0x7f5bdc017000 comp rx=0 tx=0).stop 2023-06-23T01:31:39.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5be80b00e0 msgr2=0x7f5be80c7ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5be80b00e0 0x7f5be80c7ed0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5bd800ec80 tx=0x7f5bd80091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 shutdown_connections 2023-06-23T01:31:39.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5bc8066e90 0x7f5bc8069340 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f5be80c9950 0x7f5be81484a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f5be80b00e0 0x7f5be80c7ed0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.706 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 --2- 172.21.15.92:0/3361635441 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f5be8159310 0x7f5be815f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.707 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 >> 172.21.15.92:0/3361635441 conn(0x7f5be8070290 msgr2=0x7f5be80b3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:39.707 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 shutdown_connections 2023-06-23T01:31:39.707 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.704+0000 7f5be759e700 1 -- 172.21.15.92:0/3361635441 wait complete. 2023-06-23T01:31:39.707 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'rbd' created 2023-06-23T01:31:39.717 DEBUG:teuthology.orchestra.run.smithi092:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-06-23T01:31:39.833 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.832+0000 7f29572e2700 1 Processor -- start 2023-06-23T01:31:39.834 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.832+0000 7f29572e2700 1 -- start start 2023-06-23T01:31:39.835 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.833+0000 7f29572e2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f29501595d0 0x7f29501599a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.835 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.833+0000 7f29572e2700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f295006e1c0 0x7f295006e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.835 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.833+0000 7f29572e2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29500b05d0 con 0x7f2950067410 2023-06-23T01:31:39.835 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.833+0000 7f29572e2700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f295006f4e0 con 0x7f295006e1c0 2023-06-23T01:31:39.835 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.833+0000 7f29572e2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29500caea0 con 0x7f29501595d0 2023-06-23T01:31:39.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.833+0000 7f295487d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f295006e1c0 0x7f295006e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.833+0000 7f295487d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f295006e1c0 0x7f295006e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53950/0 (socket says 172.21.15.92:53950) 2023-06-23T01:31:39.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.833+0000 7f295487d700 1 -- 172.21.15.92:0/2105452048 learned_addr learned my addr 172.21.15.92:0/2105452048 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:39.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f295507e700 1 --2- 172.21.15.92:0/2105452048 >> v2:172.21.15.119:3300/0 conn(0x7f29501595d0 0x7f29501599a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f295487d700 1 -- 172.21.15.92:0/2105452048 >> v2:172.21.15.119:3300/0 conn(0x7f29501595d0 msgr2=0x7f29501599a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:39.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f295487d700 1 --2- 172.21.15.92:0/2105452048 >> v2:172.21.15.119:3300/0 conn(0x7f29501595d0 0x7f29501599a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.836 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f295487d700 1 -- 172.21.15.92:0/2105452048 >> v1:172.21.15.119:6789/0 conn(0x7f2950067410 legacy=0x7f295015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f295487d700 1 -- 172.21.15.92:0/2105452048 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295006ec40 con 0x7f295006e1c0 2023-06-23T01:31:39.837 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f295487d700 1 --2- 172.21.15.92:0/2105452048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f295006e1c0 0x7f295006e660 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29500ce370 tx=0x7f2940009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28fce9d344ed4bb server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:39.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f2947fff700 1 -- 172.21.15.92:0/2105452048 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2940009e70 con 0x7f295006e1c0 2023-06-23T01:31:39.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f2947fff700 1 -- 172.21.15.92:0/2105452048 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f294000b860 con 0x7f295006e1c0 2023-06-23T01:31:39.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.834+0000 7f2947fff700 1 -- 172.21.15.92:0/2105452048 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f294000bc00 con 0x7f295006e1c0 2023-06-23T01:31:39.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.835+0000 7f29572e2700 1 -- 172.21.15.92:0/2105452048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f295006e1c0 msgr2=0x7f295006e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.835+0000 7f29572e2700 1 --2- 172.21.15.92:0/2105452048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f295006e1c0 0x7f295006e660 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f29500ce370 tx=0x7f2940009250 comp rx=0 tx=0).stop 2023-06-23T01:31:39.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.835+0000 7f29572e2700 1 -- 172.21.15.92:0/2105452048 shutdown_connections 2023-06-23T01:31:39.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.835+0000 7f29572e2700 1 --2- 172.21.15.92:0/2105452048 >> v2:172.21.15.119:3300/0 conn(0x7f29501595d0 0x7f29501599a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.835+0000 7f29572e2700 1 --2- 172.21.15.92:0/2105452048 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f295006e1c0 0x7f295006e660 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.835+0000 7f29572e2700 1 -- 172.21.15.92:0/2105452048 >> 172.21.15.92:0/2105452048 conn(0x7f2950068520 msgr2=0x7f2950068920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:39.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.835+0000 7f29572e2700 1 -- 172.21.15.92:0/2105452048 shutdown_connections 2023-06-23T01:31:39.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.835+0000 7f29572e2700 1 -- 172.21.15.92:0/2105452048 wait complete. 2023-06-23T01:31:39.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.836+0000 7f29572e2700 1 Processor -- start 2023-06-23T01:31:39.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.836+0000 7f29572e2700 1 -- start start 2023-06-23T01:31:39.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29572e2700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2950067410 0x7f29500ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29572e2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f295006e1c0 0x7f29500c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29572e2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29501595d0 0x7f295006a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29572e2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295016d0c0 con 0x7f29501595d0 2023-06-23T01:31:39.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29572e2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2950169fa0 con 0x7f295006e1c0 2023-06-23T01:31:39.841 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29572e2700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f295016a280 con 0x7f2950067410 2023-06-23T01:31:39.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29501595d0 0x7f295006a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29501595d0 0x7f295006a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53966/0 (socket says 172.21.15.92:53966) 2023-06-23T01:31:39.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 -- 172.21.15.92:0/1109729471 learned_addr learned my addr 172.21.15.92:0/1109729471 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:39.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295507e700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2950067410 0x7f29500ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295487d700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f295006e1c0 0x7f29500c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 -- 172.21.15.92:0/1109729471 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2950067410 msgr2=0x7f29500ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2950067410 0x7f29500ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.842 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 -- 172.21.15.92:0/1109729471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f295006e1c0 msgr2=0x7f29500c78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:39.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f295006e1c0 0x7f29500c78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:39.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 -- 172.21.15.92:0/1109729471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29400094f0 con 0x7f29501595d0 2023-06-23T01:31:39.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f295587f700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29501595d0 0x7f295006a480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f294800c840 tx=0x7f2948001670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:39.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29457fa700 1 -- 172.21.15.92:0/1109729471 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948012630 con 0x7f29501595d0 2023-06-23T01:31:39.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29457fa700 1 -- 172.21.15.92:0/1109729471 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29480127b0 con 0x7f29501595d0 2023-06-23T01:31:39.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29457fa700 1 -- 172.21.15.92:0/1109729471 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2948012b70 con 0x7f29501595d0 2023-06-23T01:31:39.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.837+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295016a560 con 0x7f29501595d0 2023-06-23T01:31:39.844 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.838+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29500aef10 con 0x7f29501595d0 2023-06-23T01:31:39.844 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.838+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29500af1c0 con 0x7f29501595d0 2023-06-23T01:31:39.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.839+0000 7f29457fa700 1 -- 172.21.15.92:0/1109729471 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f294801a3a0 con 0x7f29501595d0 2023-06-23T01:31:39.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.839+0000 7f29457fa700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2934066e90 0x7f2934069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:39.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.839+0000 7f29457fa700 1 -- 172.21.15.92:0/1109729471 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5195+0+0 (secure 0 0 0) 0x7f2948084390 con 0x7f29501595d0 2023-06-23T01:31:39.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.839+0000 7f295507e700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2934066e90 0x7f2934069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:39.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.840+0000 7f295507e700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2934066e90 0x7f2934069340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f294c009db0 tx=0x7f294c0091a0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:39.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:39.844+0000 7f29457fa700 1 -- 172.21.15.92:0/1109729471 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f29500af1c0 con 0x7f29501595d0 2023-06-23T01:31:40.027 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.026+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f295016d230 con 0x7f29501595d0 2023-06-23T01:31:40.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.710+0000 7f29457fa700 1 -- 172.21.15.92:0/1109729471 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f295016d230 con 0x7f29501595d0 2023-06-23T01:31:40.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.713+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2934066e90 msgr2=0x7f2934069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:40.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.713+0000 7f29572e2700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2934066e90 0x7f2934069340 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f294c009db0 tx=0x7f294c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.713+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29501595d0 msgr2=0x7f295006a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:40.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.713+0000 7f29572e2700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29501595d0 0x7f295006a480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f294800c840 tx=0x7f2948001670 comp rx=0 tx=0).stop 2023-06-23T01:31:40.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.714+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 shutdown_connections 2023-06-23T01:31:40.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.714+0000 7f29572e2700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2934066e90 0x7f2934069340 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.714+0000 7f29572e2700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f29501595d0 0x7f295006a480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.714+0000 7f29572e2700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2950067410 0x7f29500ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.714+0000 7f29572e2700 1 --2- 172.21.15.92:0/1109729471 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f295006e1c0 0x7f29500c78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.714+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 >> 172.21.15.92:0/1109729471 conn(0x7f2950068520 msgr2=0x7f29500b1750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:40.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.714+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 shutdown_connections 2023-06-23T01:31:40.717 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.714+0000 7f29572e2700 1 -- 172.21.15.92:0/1109729471 wait complete. 2023-06-23T01:31:40.717 INFO:teuthology.orchestra.run.smithi092.stderr:enabled application 'rbd' on pool 'rbd' 2023-06-23T01:31:40.728 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-06-23T01:31:40.729 INFO:tasks.ceph.mds.a:Restarting daemon 2023-06-23T01:31:40.729 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-06-23T01:31:40.731 INFO:tasks.ceph.mds.a:Started 2023-06-23T01:31:40.732 INFO:tasks.ceph.mds.c:Restarting daemon 2023-06-23T01:31:40.732 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-06-23T01:31:40.735 INFO:tasks.ceph.mds.c:Started 2023-06-23T01:31:40.735 INFO:tasks.ceph.mds.b:Restarting daemon 2023-06-23T01:31:40.735 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-06-23T01:31:40.738 INFO:tasks.ceph.mds.b:Started 2023-06-23T01:31:40.739 INFO:tasks.ceph.mds.d:Restarting daemon 2023-06-23T01:31:40.739 DEBUG:teuthology.orchestra.run.smithi119:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-06-23T01:31:40.741 INFO:tasks.ceph.mds.d:Started 2023-06-23T01:31:40.742 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-06-23T01:31:40.742 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:31:40.806 INFO:tasks.ceph.mds.a.smithi092.stdout:starting mds.a at 2023-06-23T01:31:40.807 INFO:tasks.ceph.mds.c.smithi092.stdout:starting mds.c at 2023-06-23T01:31:40.821 INFO:tasks.ceph.mds.b.smithi119.stdout:starting mds.b at 2023-06-23T01:31:40.823 INFO:tasks.ceph.mds.d.smithi119.stdout:starting mds.d at 2023-06-23T01:31:40.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.864+0000 7f9bc8b84700 1 Processor -- start 2023-06-23T01:31:40.866 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.864+0000 7f9bc8b84700 1 -- start start 2023-06-23T01:31:40.867 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc8b84700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bc4159430 0x7f9bc4159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:40.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc8b84700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40cbd60 0x7f9bc40cd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:40.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc8b84700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc40ccec0 con 0x7f9bc40b2790 2023-06-23T01:31:40.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc8b84700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc40cc2b0 con 0x7f9bc4159430 2023-06-23T01:31:40.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc8b84700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40cc590 con 0x7f9bc40cbd60 2023-06-23T01:31:40.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc259c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bc4159430 0x7f9bc4159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:40.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc2d9d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f9bc40b2790 0x7f9bc4068380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:41028/0 (socket says 172.21.15.92:41028) 2023-06-23T01:31:40.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc259c700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f9bc4159430 0x7f9bc4159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:53998/0 (socket says 172.21.15.92:53998) 2023-06-23T01:31:40.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc2d9d700 1 -- 172.21.15.92:0/3984805395 learned_addr learned my addr 172.21.15.92:0/3984805395 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:40.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc1d9b700 1 --2- 172.21.15.92:0/3984805395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40cbd60 0x7f9bc40cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:40.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc259c700 1 -- 172.21.15.92:0/3984805395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40cbd60 msgr2=0x7f9bc40cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:40.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc259c700 1 --2- 172.21.15.92:0/3984805395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40cbd60 0x7f9bc40cd350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc259c700 1 -- 172.21.15.92:0/3984805395 >> v1:172.21.15.92:6789/0 conn(0x7f9bc40b2790 legacy=0x7f9bc4068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:40.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc259c700 1 -- 172.21.15.92:0/3984805395 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc40cc870 con 0x7f9bc4159430 2023-06-23T01:31:40.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc159a700 1 -- 172.21.15.92:0/3984805395 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235835167 0 0) 0x7f9bc40ccec0 con 0x7f9bc40b2790 2023-06-23T01:31:40.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc259c700 1 --2- 172.21.15.92:0/3984805395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc4159800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40098e0 tx=0x7f9bb40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3bfc36f3673a45d server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:40.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc159a700 1 -- 172.21.15.92:0/3984805395 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400b640 con 0x7f9bc4159430 2023-06-23T01:31:40.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc159a700 1 -- 172.21.15.92:0/3984805395 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb400b7a0 con 0x7f9bc4159430 2023-06-23T01:31:40.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.865+0000 7f9bc159a700 1 -- 172.21.15.92:0/3984805395 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb400bb40 con 0x7f9bc4159430 2023-06-23T01:31:40.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.866+0000 7f9bc8b84700 1 -- 172.21.15.92:0/3984805395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 msgr2=0x7f9bc4159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:40.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.866+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/3984805395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc4159800 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40098e0 tx=0x7f9bb40091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.866+0000 7f9bc8b84700 1 -- 172.21.15.92:0/3984805395 shutdown_connections 2023-06-23T01:31:40.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.866+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/3984805395 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc4159800 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.866+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/3984805395 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40cbd60 0x7f9bc40cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.866+0000 7f9bc8b84700 1 -- 172.21.15.92:0/3984805395 >> 172.21.15.92:0/3984805395 conn(0x7f9bc4070290 msgr2=0x7f9bc4070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:40.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.866+0000 7f9bc8b84700 1 -- 172.21.15.92:0/3984805395 shutdown_connections 2023-06-23T01:31:40.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.866+0000 7f9bc8b84700 1 -- 172.21.15.92:0/3984805395 wait complete. 2023-06-23T01:31:40.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.867+0000 7f9bc8b84700 1 Processor -- start 2023-06-23T01:31:40.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.867+0000 7f9bc8b84700 1 -- start start 2023-06-23T01:31:40.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc8b84700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40b2790 0x7f9bc40ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:40.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc8b84700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bc40cbd60 0x7f9bc4169fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:40.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc8b84700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc40c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:40.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc8b84700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40ce0c0 con 0x7f9bc4159430 2023-06-23T01:31:40.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc8b84700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc40ce680 con 0x7f9bc40cbd60 2023-06-23T01:31:40.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc8b84700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc416aa90 con 0x7f9bc40b2790 2023-06-23T01:31:40.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc40c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:40.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc2d9d700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc40c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54000/0 (socket says 172.21.15.92:54000) 2023-06-23T01:31:40.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc2d9d700 1 -- 172.21.15.92:0/2901380038 learned_addr learned my addr 172.21.15.92:0/2901380038 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:40.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc259c700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40b2790 0x7f9bc40ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:40.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc1d9b700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bc40cbd60 0x7f9bc4169fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:40.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc2d9d700 1 -- 172.21.15.92:0/2901380038 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40b2790 msgr2=0x7f9bc40ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:40.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc2d9d700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40b2790 0x7f9bc40ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc2d9d700 1 -- 172.21.15.92:0/2901380038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bc40cbd60 msgr2=0x7f9bc4169fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:40.875 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc2d9d700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bc40cbd60 0x7f9bc4169fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:40.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.868+0000 7f9bc2d9d700 1 -- 172.21.15.92:0/2901380038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb4009420 con 0x7f9bc4159430 2023-06-23T01:31:40.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.869+0000 7f9bc2d9d700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc40c55a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00cc90 tx=0x7f9bbc001a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:40.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.869+0000 7f9bb27fc700 1 -- 172.21.15.92:0/2901380038 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc00a870 con 0x7f9bc4159430 2023-06-23T01:31:40.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.869+0000 7f9bb27fc700 1 -- 172.21.15.92:0/2901380038 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bbc00a9d0 con 0x7f9bc4159430 2023-06-23T01:31:40.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.869+0000 7f9bb27fc700 1 -- 172.21.15.92:0/2901380038 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bbc00ad90 con 0x7f9bc4159430 2023-06-23T01:31:40.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.869+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc416ad70 con 0x7f9bc4159430 2023-06-23T01:31:40.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.869+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc40cf050 con 0x7f9bc4159430 2023-06-23T01:31:40.876 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.870+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc40ce7f0 con 0x7f9bc4159430 2023-06-23T01:31:40.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.870+0000 7f9bb27fc700 1 -- 172.21.15.92:0/2901380038 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f9bbc007300 con 0x7f9bc4159430 2023-06-23T01:31:40.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.870+0000 7f9bb27fc700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f9ba4066e90 0x7f9ba4069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:40.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.871+0000 7f9bc259c700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f9ba4066e90 0x7f9ba4069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:40.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.871+0000 7f9bb27fc700 1 -- 172.21.15.92:0/2901380038 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f9bbc0845b0 con 0x7f9bc4159430 2023-06-23T01:31:40.877 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.871+0000 7f9bc259c700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f9ba4066e90 0x7f9ba4069340 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40091e0 tx=0x7f9bb4017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:40.878 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:40.875+0000 7f9bb27fc700 1 -- 172.21.15.92:0/2901380038 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f9bbc051b50 con 0x7f9bc4159430 2023-06-23T01:31:41.062 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.060+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9bc404c1b0 con 0x7f9bc4159430 2023-06-23T01:31:41.063 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.061+0000 7f9bb27fc700 1 -- 172.21.15.92:0/2901380038 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+13660 (secure 0 0 0) 0x7f9bbc055170 con 0x7f9bc4159430 2023-06-23T01:31:41.063 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:31:41.063 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":18,"fsid":"c94811cf-f7e5-4c57-b505-a983532f8da8","created":"2023-06-23T01:31:20.907484+0000","modified":"2023-06-23T01:31:40.700568+0000","last_up_change":"2023-06-23T01:31:32.355093+0000","last_in_change":"2023-06-23T01:31:23.354162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:31:32.414191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:31:39.216467+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4bcf4cdc-7c98-4a4a-bc56-133f22f4d802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":87954},{"type":"v1","addr":"172.21.15.92:6809","nonce":87954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":87954},{"type":"v1","addr":"172.21.15.92:6811","nonce":87954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":87954},{"type":"v1","addr":"172.21.15.92:6815","nonce":87954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":87954},{"type":"v1","addr":"172.21.15.92:6813","nonce":87954}]},"public_addr":"172.21.15.92:6809/87954","cluster_addr":"172.21.15.92:6811/87954","heartbeat_back_addr":"172.21.15.92:6815/87954","heartbeat_front_addr":"172.21.15.92:6813/87954","state":["exists","up"]},{"osd":1,"uuid":"a2e05a19-0c70-4cb4-aacd-3381219435b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":87955},{"type":"v1","addr":"172.21.15.92:6821","nonce":87955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":87955},{"type":"v1","addr":"172.21.15.92:6826","nonce":87955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":87955},{"type":"v1","addr":"172.21.15.92:6831","nonce":87955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":87955},{"type":"v1","addr":"172.21.15.92:6829","nonce":87955}]},"public_addr":"172.21.15.92:6821/87955","cluster_addr":"172.21.15.92:6826/87955","heartbeat_back_addr":"172.21.15.92:6831/87955","heartbeat_front_addr":"172.21.15.92:6829/87955","state":["exists","up"]},{"osd":2,"uuid":"20f85617-68ff-491f-a436-aa1ad1979595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":87951},{"type":"v1","addr":"172.21.15.92:6801","nonce":87951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":87951},{"type":"v1","addr":"172.21.15.92:6803","nonce":87951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":87951},{"type":"v1","addr":"172.21.15.92:6807","nonce":87951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":87951},{"type":"v1","addr":"172.21.15.92:6805","nonce":87951}]},"public_addr":"172.21.15.92:6801/87951","cluster_addr":"172.21.15.92:6803/87951","heartbeat_back_addr":"172.21.15.92:6807/87951","heartbeat_front_addr":"172.21.15.92:6805/87951","state":["exists","up"]},{"osd":3,"uuid":"3e5b05f4-6b77-4612-9720-38ece0b1388b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":87956},{"type":"v1","addr":"172.21.15.92:6817","nonce":87956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":87956},{"type":"v1","addr":"172.21.15.92:6820","nonce":87956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":87956},{"type":"v1","addr":"172.21.15.92:6827","nonce":87956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":87956},{"type":"v1","addr":"172.21.15.92:6824","nonce":87956}]},"public_addr":"172.21.15.92:6817/87956","cluster_addr":"172.21.15.92:6820/87956","heartbeat_back_addr":"172.21.15.92:6827/87956","heartbeat_front_addr":"172.21.15.92:6824/87956","state":["exists","up"]},{"osd":4,"uuid":"4d94d9c9-4288-477f-801f-c66683295335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6801","nonce":88172},{"type":"v1","addr":"172.21.15.119:6802","nonce":88172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":88172},{"type":"v1","addr":"172.21.15.119:6806","nonce":88172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":88172},{"type":"v1","addr":"172.21.15.119:6814","nonce":88172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":88172},{"type":"v1","addr":"172.21.15.119:6810","nonce":88172}]},"public_addr":"172.21.15.119:6802/88172","cluster_addr":"172.21.15.119:6806/88172","heartbeat_back_addr":"172.21.15.119:6814/88172","heartbeat_front_addr":"172.21.15.119:6810/88172","state":["exists","up"]},{"osd":5,"uuid":"dc6b2f59-8af4-42b2-b7f9-bf36714c358f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":88173},{"type":"v1","addr":"172.21.15.119:6803","nonce":88173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6805","nonce":88173},{"type":"v1","addr":"172.21.15.119:6807","nonce":88173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":88173},{"type":"v1","addr":"172.21.15.119:6815","nonce":88173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":88173},{"type":"v1","addr":"172.21.15.119:6811","nonce":88173}]},"public_addr":"172.21.15.119:6803/88173","cluster_addr":"172.21.15.119:6807/88173","heartbeat_back_addr":"172.21.15.119:6815/88173","heartbeat_front_addr":"172.21.15.119:6811/88173","state":["exists","up"]},{"osd":6,"uuid":"2ac0eb61-39bf-4666-8c24-279bf8b9c51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6817","nonce":88202},{"type":"v1","addr":"172.21.15.119:6819","nonce":88202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6821","nonce":88202},{"type":"v1","addr":"172.21.15.119:6823","nonce":88202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6829","nonce":88202},{"type":"v1","addr":"172.21.15.119:6831","nonce":88202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6825","nonce":88202},{"type":"v1","addr":"172.21.15.119:6827","nonce":88202}]},"public_addr":"172.21.15.119:6819/88202","cluster_addr":"172.21.15.119:6823/88202","heartbeat_back_addr":"172.21.15.119:6831/88202","heartbeat_front_addr":"172.21.15.119:6827/88202","state":["exists","up"]},{"osd":7,"uuid":"c4ff77b0-23dc-477d-adaa-d7fd9108f5da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":88203},{"type":"v1","addr":"172.21.15.119:6818","nonce":88203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":88203},{"type":"v1","addr":"172.21.15.119:6822","nonce":88203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":88203},{"type":"v1","addr":"172.21.15.119:6830","nonce":88203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":88203},{"type":"v1","addr":"172.21.15.119:6826","nonce":88203}]},"public_addr":"172.21.15.119:6818/88203","cluster_addr":"172.21.15.119:6822/88203","heartbeat_back_addr":"172.21.15.119:6830/88203","heartbeat_front_addr":"172.21.15.119:6826/88203","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.542554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.495262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.517893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.443537+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.444113+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.472890+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.556249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.573872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:31:41.065 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f9ba4066e90 msgr2=0x7f9ba4069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.065 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f9ba4066e90 0x7f9ba4069340 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40091e0 tx=0x7f9bb4017000 comp rx=0 tx=0).stop 2023-06-23T01:31:41.065 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 msgr2=0x7f9bc40c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.065 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc40c55a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f9bbc00cc90 tx=0x7f9bbc001a20 comp rx=0 tx=0).stop 2023-06-23T01:31:41.065 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 shutdown_connections 2023-06-23T01:31:41.066 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f9ba4066e90 0x7f9ba4069340 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.066 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f9bc4159430 0x7f9bc40c55a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.066 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f9bc40b2790 0x7f9bc40ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.066 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 --2- 172.21.15.92:0/2901380038 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f9bc40cbd60 0x7f9bc4169fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.066 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.063+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 >> 172.21.15.92:0/2901380038 conn(0x7f9bc4070290 msgr2=0x7f9bc40c8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:41.066 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.064+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 shutdown_connections 2023-06-23T01:31:41.066 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.064+0000 7f9bc8b84700 1 -- 172.21.15.92:0/2901380038 wait complete. 2023-06-23T01:31:41.077 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:31:32.414191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-23T01:31:39.216467+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-06-23T01:31:41.077 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:31:41.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.193+0000 7f2698f09700 1 Processor -- start 2023-06-23T01:31:41.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.193+0000 7f2698f09700 1 -- start start 2023-06-23T01:31:41.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2698f09700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f26940d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2698f09700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f26940b2010 0x7f26940b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.197 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2698f09700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26940cedd0 con 0x7f2694153a80 2023-06-23T01:31:41.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2698f09700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26940d14c0 con 0x7f26940b2010 2023-06-23T01:31:41.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2698f09700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26940b29d0 con 0x7f2694153e50 2023-06-23T01:31:41.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2689d9b700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f26940d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2692d9d700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f26940b2010 0x7f26940b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2689d9b700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f26940d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.92:46584/0 (socket says 172.21.15.92:46584) 2023-06-23T01:31:41.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f2689d9b700 1 -- 172.21.15.92:0/1712748455 learned_addr learned my addr 172.21.15.92:0/1712748455 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:41.198 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.194+0000 7f269259c700 1 --1- 172.21.15.92:0/1712748455 >> v1:172.21.15.119:6789/0 conn(0x7f2694153a80 0x7f269415a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.92:39662/0 (socket says 172.21.15.92:39662) 2023-06-23T01:31:41.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2689d9b700 1 -- 172.21.15.92:0/1712748455 >> v2:172.21.15.119:3300/0 conn(0x7f26940b2010 msgr2=0x7f26940b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:41.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2689d9b700 1 --2- 172.21.15.92:0/1712748455 >> v2:172.21.15.119:3300/0 conn(0x7f26940b2010 0x7f26940b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2689d9b700 1 -- 172.21.15.92:0/1712748455 >> v1:172.21.15.119:6789/0 conn(0x7f2694153a80 legacy=0x7f269415a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2689d9b700 1 -- 172.21.15.92:0/1712748455 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f269415aed0 con 0x7f2694153e50 2023-06-23T01:31:41.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2689d9b700 1 --2- 172.21.15.92:0/1712748455 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f26940d0590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f267c0099b0 tx=0x7f267c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90ddd2aee2f0d3ba server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2691d9b700 1 -- 172.21.15.92:0/1712748455 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c009ea0 con 0x7f2694153e50 2023-06-23T01:31:41.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2691d9b700 1 -- 172.21.15.92:0/1712748455 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c00b860 con 0x7f2694153e50 2023-06-23T01:31:41.199 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2691d9b700 1 -- 172.21.15.92:0/1712748455 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c00bc00 con 0x7f2694153e50 2023-06-23T01:31:41.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2698f09700 1 -- 172.21.15.92:0/1712748455 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 msgr2=0x7f26940d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2698f09700 1 --2- 172.21.15.92:0/1712748455 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f26940d0590 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f267c0099b0 tx=0x7f267c009250 comp rx=0 tx=0).stop 2023-06-23T01:31:41.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2698f09700 1 -- 172.21.15.92:0/1712748455 shutdown_connections 2023-06-23T01:31:41.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2698f09700 1 --2- 172.21.15.92:0/1712748455 >> v2:172.21.15.119:3300/0 conn(0x7f26940b2010 0x7f26940b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2698f09700 1 --2- 172.21.15.92:0/1712748455 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f26940d0590 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.195+0000 7f2698f09700 1 -- 172.21.15.92:0/1712748455 >> 172.21.15.92:0/1712748455 conn(0x7f2694070290 msgr2=0x7f2694070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:41.200 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.196+0000 7f2698f09700 1 -- 172.21.15.92:0/1712748455 shutdown_connections 2023-06-23T01:31:41.201 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.196+0000 7f2698f09700 1 -- 172.21.15.92:0/1712748455 wait complete. 2023-06-23T01:31:41.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.197+0000 7f2698f09700 1 Processor -- start 2023-06-23T01:31:41.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.197+0000 7f2698f09700 1 -- start start 2023-06-23T01:31:41.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.197+0000 7f2698f09700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26940b2010 0x7f26941564a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.197+0000 7f2698f09700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2694153a80 0x7f2694158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.203 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.197+0000 7f2698f09700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f2694145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.197+0000 7f2698f09700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694157750 con 0x7f2694153a80 2023-06-23T01:31:41.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.197+0000 7f2698f09700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2694157be0 con 0x7f26940b2010 2023-06-23T01:31:41.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.197+0000 7f2698f09700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2694156b40 con 0x7f2694153e50 2023-06-23T01:31:41.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2694153a80 0x7f2694158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f269259c700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26940b2010 0x7f26941564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2694153a80 0x7f2694158a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:54014/0 (socket says 172.21.15.92:54014) 2023-06-23T01:31:41.204 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 -- 172.21.15.92:0/2777919800 learned_addr learned my addr 172.21.15.92:0/2777919800 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:41.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2692d9d700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f2694145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 -- 172.21.15.92:0/2777919800 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 msgr2=0x7f2694145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f2694145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 -- 172.21.15.92:0/2777919800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26940b2010 msgr2=0x7f26941564a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26940b2010 0x7f26941564a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 -- 172.21.15.92:0/2777919800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267c0094f0 con 0x7f2694153a80 2023-06-23T01:31:41.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f269259c700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26940b2010 0x7f26941564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:41.205 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2689d9b700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2694153a80 0x7f2694158a40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f267c006fd0 tx=0x7f267c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f268affd700 1 -- 172.21.15.92:0/2777919800 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c000d10 con 0x7f2694153a80 2023-06-23T01:31:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f268affd700 1 -- 172.21.15.92:0/2777919800 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f267c00bbe0 con 0x7f2694153a80 2023-06-23T01:31:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2694156e20 con 0x7f2694153a80 2023-06-23T01:31:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f268affd700 1 -- 172.21.15.92:0/2777919800 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f267c025670 con 0x7f2694153a80 2023-06-23T01:31:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.198+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26941573e0 con 0x7f2694153a80 2023-06-23T01:31:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.199+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26940892f0 con 0x7f2694153a80 2023-06-23T01:31:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.199+0000 7f268affd700 1 -- 172.21.15.92:0/2777919800 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f267c02e440 con 0x7f2694153a80 2023-06-23T01:31:41.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.200+0000 7f268affd700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2674066e90 0x7f2674069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.200+0000 7f268affd700 1 -- 172.21.15.92:0/2777919800 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f267c085ec0 con 0x7f2694153a80 2023-06-23T01:31:41.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.200+0000 7f269259c700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2674066e90 0x7f2674069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.200+0000 7f269259c700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2674066e90 0x7f2674069340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2684001710 tx=0x7f2684007eb0 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.207 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.204+0000 7f268affd700 1 -- 172.21.15.92:0/2777919800 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f267c0533e0 con 0x7f2694153a80 2023-06-23T01:31:41.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.385+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f269404c1b0 con 0x7f2694153a80 2023-06-23T01:31:41.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.386+0000 7f268affd700 1 -- 172.21.15.92:0/2777919800 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7f267c056a00 con 0x7f2694153a80 2023-06-23T01:31:41.388 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2023-06-23T01:31:41.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.387+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2674066e90 msgr2=0x7f2674069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.387+0000 7f2698f09700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2674066e90 0x7f2674069340 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2684001710 tx=0x7f2684007eb0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.387+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2694153a80 msgr2=0x7f2694158a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.387+0000 7f2698f09700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2694153a80 0x7f2694158a40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f267c006fd0 tx=0x7f267c017000 comp rx=0 tx=0).stop 2023-06-23T01:31:41.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.387+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 shutdown_connections 2023-06-23T01:31:41.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.387+0000 7f2698f09700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2674066e90 0x7f2674069340 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.387+0000 7f2698f09700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2694153e50 0x7f2694145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.387+0000 7f2698f09700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f26940b2010 0x7f26941564a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.388+0000 7f2698f09700 1 --2- 172.21.15.92:0/2777919800 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2694153a80 0x7f2694158a40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.388+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 >> 172.21.15.92:0/2777919800 conn(0x7f2694070290 msgr2=0x7f269415b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:41.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.388+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 shutdown_connections 2023-06-23T01:31:41.391 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.388+0000 7f2698f09700 1 -- 172.21.15.92:0/2777919800 wait complete. 2023-06-23T01:31:41.401 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T01:31:41.517 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.516+0000 7f2b4125f700 1 Processor -- start 2023-06-23T01:31:41.517 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.516+0000 7f2b4125f700 1 -- start start 2023-06-23T01:31:41.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.516+0000 7f2b4125f700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c153e50 0x7f2b3c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.516+0000 7f2b4125f700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2b3c0b2010 0x7f2b3c0b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.516+0000 7f2b4125f700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b3c0cedd0 con 0x7f2b3c153a80 2023-06-23T01:31:41.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.516+0000 7f2b4125f700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b3c0d14c0 con 0x7f2b3c0b2010 2023-06-23T01:31:41.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.516+0000 7f2b4125f700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0b29d0 con 0x7f2b3c153e50 2023-06-23T01:31:41.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3b59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f2b3c0b2010 0x7f2b3c0b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3ad9d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2b3c153a80 0x7f2b3c15a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36086/0 (socket says 172.21.15.92:36086) 2023-06-23T01:31:41.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3ad9d700 1 -- 172.21.15.92:0/87575228 learned_addr learned my addr 172.21.15.92:0/87575228 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:41.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3259c700 1 --2- 172.21.15.92:0/87575228 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c153e50 0x7f2b3c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3b59e700 1 -- 172.21.15.92:0/87575228 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c153e50 msgr2=0x7f2b3c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:41.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3b59e700 1 --2- 172.21.15.92:0/87575228 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c153e50 0x7f2b3c0d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3b59e700 1 -- 172.21.15.92:0/87575228 >> v1:172.21.15.92:6789/0 conn(0x7f2b3c153a80 legacy=0x7f2b3c15a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3a59c700 1 -- 172.21.15.92:0/87575228 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205435157 0 0) 0x7f2b3c0cedd0 con 0x7f2b3c153a80 2023-06-23T01:31:41.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3b59e700 1 -- 172.21.15.92:0/87575228 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b3c15aed0 con 0x7f2b3c0b2010 2023-06-23T01:31:41.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3259c700 1 --2- 172.21.15.92:0/87575228 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c153e50 0x7f2b3c0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:31:41.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3b59e700 1 --2- 172.21.15.92:0/87575228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c0b2010 0x7f2b3c0b2490 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009e90 tx=0x7f2b34009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb77ebfbee6fee83 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3a59c700 1 -- 172.21.15.92:0/87575228 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3400e070 con 0x7f2b3c0b2010 2023-06-23T01:31:41.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3a59c700 1 -- 172.21.15.92:0/87575228 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3400bbf0 con 0x7f2b3c0b2010 2023-06-23T01:31:41.523 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.517+0000 7f2b3a59c700 1 -- 172.21.15.92:0/87575228 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34012430 con 0x7f2b3c0b2010 2023-06-23T01:31:41.523 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.518+0000 7f2b4125f700 1 -- 172.21.15.92:0/87575228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c0b2010 msgr2=0x7f2b3c0b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.523 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.518+0000 7f2b4125f700 1 --2- 172.21.15.92:0/87575228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c0b2010 0x7f2b3c0b2490 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2b34009e90 tx=0x7f2b34009780 comp rx=0 tx=0).stop 2023-06-23T01:31:41.523 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.518+0000 7f2b4125f700 1 -- 172.21.15.92:0/87575228 shutdown_connections 2023-06-23T01:31:41.523 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.518+0000 7f2b4125f700 1 --2- 172.21.15.92:0/87575228 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c0b2010 0x7f2b3c0b2490 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.523 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.518+0000 7f2b4125f700 1 --2- 172.21.15.92:0/87575228 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c153e50 0x7f2b3c0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.523 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.518+0000 7f2b4125f700 1 -- 172.21.15.92:0/87575228 >> 172.21.15.92:0/87575228 conn(0x7f2b3c070290 msgr2=0x7f2b3c070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:41.524 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.518+0000 7f2b4125f700 1 -- 172.21.15.92:0/87575228 shutdown_connections 2023-06-23T01:31:41.524 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.518+0000 7f2b4125f700 1 -- 172.21.15.92:0/87575228 wait complete. 2023-06-23T01:31:41.526 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.519+0000 7f2b4125f700 1 Processor -- start 2023-06-23T01:31:41.526 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.519+0000 7f2b4125f700 1 -- start start 2023-06-23T01:31:41.526 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b4125f700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c0b2010 0x7f2b3c0b59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.526 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b4125f700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b3c153a80 0x7f2b3c156720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.526 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b4125f700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c153e50 0x7f2b3c169760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.526 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b4125f700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c0b5520 con 0x7f2b3c153e50 2023-06-23T01:31:41.527 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b4125f700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c156c60 con 0x7f2b3c153a80 2023-06-23T01:31:41.527 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b4125f700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b3c153680 con 0x7f2b3c0b2010 2023-06-23T01:31:41.527 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c153e50 0x7f2b3c169760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.527 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c153e50 0x7f2b3c169760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57730/0 (socket says 172.21.15.92:57730) 2023-06-23T01:31:41.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 -- 172.21.15.92:0/3417112253 learned_addr learned my addr 172.21.15.92:0/3417112253 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:41.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3ad9d700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c0b2010 0x7f2b3c0b59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3259c700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b3c153a80 0x7f2b3c156720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 -- 172.21.15.92:0/3417112253 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c0b2010 msgr2=0x7f2b3c0b59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c0b2010 0x7f2b3c0b59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 -- 172.21.15.92:0/3417112253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b3c153a80 msgr2=0x7f2b3c156720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b3c153a80 0x7f2b3c156720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.528 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 -- 172.21.15.92:0/3417112253 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b34009a20 con 0x7f2b3c153e50 2023-06-23T01:31:41.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b3b59e700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c153e50 0x7f2b3c169760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c089000 tx=0x7f2b34019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b337fe700 1 -- 172.21.15.92:0/3417112253 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b34012e70 con 0x7f2b3c153e50 2023-06-23T01:31:41.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b337fe700 1 -- 172.21.15.92:0/3417112253 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b3401d630 con 0x7f2b3c153e50 2023-06-23T01:31:41.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b3c0b4910 con 0x7f2b3c153e50 2023-06-23T01:31:41.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b337fe700 1 -- 172.21.15.92:0/3417112253 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b3401d9f0 con 0x7f2b3c153e50 2023-06-23T01:31:41.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.520+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b3c0b4ed0 con 0x7f2b3c153e50 2023-06-23T01:31:41.529 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.521+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b3c0b5690 con 0x7f2b3c153e50 2023-06-23T01:31:41.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.522+0000 7f2b337fe700 1 -- 172.21.15.92:0/3417112253 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f2b34024070 con 0x7f2b3c153e50 2023-06-23T01:31:41.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.522+0000 7f2b337fe700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2b1c066e90 0x7f2b1c069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.522+0000 7f2b3ad9d700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2b1c066e90 0x7f2b1c069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.522+0000 7f2b337fe700 1 -- 172.21.15.92:0/3417112253 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f2b340869d0 con 0x7f2b3c153e50 2023-06-23T01:31:41.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.522+0000 7f2b3ad9d700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2b1c066e90 0x7f2b1c069340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0af840 tx=0x7f2b2c007f50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.530 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.526+0000 7f2b337fe700 1 -- 172.21.15.92:0/3417112253 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f2b34053fa0 con 0x7f2b3c153e50 2023-06-23T01:31:41.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.710+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2b3c04c1b0 con 0x7f2b3c153e50 2023-06-23T01:31:41.712 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.711+0000 7f2b337fe700 1 -- 172.21.15.92:0/3417112253 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2b340575c0 con 0x7f2b3c153e50 2023-06-23T01:31:41.713 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 8 2023-06-23T01:31:41.714 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2b1c066e90 msgr2=0x7f2b1c069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2b1c066e90 0x7f2b1c069340 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c0af840 tx=0x7f2b2c007f50 comp rx=0 tx=0).stop 2023-06-23T01:31:41.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c153e50 msgr2=0x7f2b3c169760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c153e50 0x7f2b3c169760 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2b3c089000 tx=0x7f2b34019000 comp rx=0 tx=0).stop 2023-06-23T01:31:41.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 shutdown_connections 2023-06-23T01:31:41.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2b1c066e90 0x7f2b1c069340 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.715 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2b3c153e50 0x7f2b3c169760 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2b3c0b2010 0x7f2b3c0b59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 --2- 172.21.15.92:0/3417112253 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2b3c153a80 0x7f2b3c156720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 >> 172.21.15.92:0/3417112253 conn(0x7f2b3c070290 msgr2=0x7f2b3c15b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:41.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 shutdown_connections 2023-06-23T01:31:41.716 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.713+0000 7f2b4125f700 1 -- 172.21.15.92:0/3417112253 wait complete. 2023-06-23T01:31:41.727 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-06-23T01:31:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.841+0000 7f3eb93f2700 1 Processor -- start 2023-06-23T01:31:41.843 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.841+0000 7f3eb93f2700 1 -- start start 2023-06-23T01:31:41.844 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.842+0000 7f3eb93f2700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb40b1030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.842+0000 7f3eb93f2700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3eb40c9770 0x7f3eb40c4350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.842+0000 7f3eb93f2700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eb40c3ec0 con 0x7f3eb40c4890 2023-06-23T01:31:41.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.842+0000 7f3eb93f2700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eb40c3010 con 0x7f3eb40c9770 2023-06-23T01:31:41.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.842+0000 7f3eb93f2700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb40c32b0 con 0x7f3eb40c93a0 2023-06-23T01:31:41.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.842+0000 7f3eb3fff700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb40b1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.845 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.842+0000 7f3eb37fe700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f3eb40c9770 0x7f3eb40c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb8bf1700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f3eb40c4890 0x7f3eb415b390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.92:38060/0 (socket says 172.21.15.92:38060) 2023-06-23T01:31:41.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb3fff700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb40b1030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.92:47900/0 (socket says 172.21.15.92:47900) 2023-06-23T01:31:41.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb8bf1700 1 -- 172.21.15.92:0/995643284 learned_addr learned my addr 172.21.15.92:0/995643284 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:41.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb37fe700 1 -- 172.21.15.92:0/995643284 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 msgr2=0x7f3eb40b1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:41.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb37fe700 1 --2- 172.21.15.92:0/995643284 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb40b1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb37fe700 1 -- 172.21.15.92:0/995643284 >> v1:172.21.15.119:6789/0 conn(0x7f3eb40c4890 legacy=0x7f3eb415b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb37fe700 1 -- 172.21.15.92:0/995643284 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb40c3590 con 0x7f3eb40c9770 2023-06-23T01:31:41.846 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb3fff700 1 --2- 172.21.15.92:0/995643284 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb40b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:41.847 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb37fe700 1 --2- 172.21.15.92:0/995643284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c9770 0x7f3eb40c4350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80098e0 tx=0x7f3ea80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c5f2abae66069cf server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.847 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb2ffd700 1 -- 172.21.15.92:0/995643284 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea800b6d0 con 0x7f3eb40c9770 2023-06-23T01:31:41.847 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb2ffd700 1 -- 172.21.15.92:0/995643284 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ea800b830 con 0x7f3eb40c9770 2023-06-23T01:31:41.847 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.843+0000 7f3eb2ffd700 1 -- 172.21.15.92:0/995643284 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ea800bbd0 con 0x7f3eb40c9770 2023-06-23T01:31:41.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.844+0000 7f3eb93f2700 1 -- 172.21.15.92:0/995643284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c9770 msgr2=0x7f3eb40c4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.849 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.844+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/995643284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c9770 0x7f3eb40c4350 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80098e0 tx=0x7f3ea80091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.844+0000 7f3eb93f2700 1 -- 172.21.15.92:0/995643284 shutdown_connections 2023-06-23T01:31:41.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.844+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/995643284 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb40b1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.844+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/995643284 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c9770 0x7f3eb40c4350 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.844+0000 7f3eb93f2700 1 -- 172.21.15.92:0/995643284 >> 172.21.15.92:0/995643284 conn(0x7f3eb4067bf0 msgr2=0x7f3eb40ce270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:41.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.844+0000 7f3eb93f2700 1 -- 172.21.15.92:0/995643284 shutdown_connections 2023-06-23T01:31:41.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.844+0000 7f3eb93f2700 1 -- 172.21.15.92:0/995643284 wait complete. 2023-06-23T01:31:41.850 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.845+0000 7f3eb93f2700 1 Processor -- start 2023-06-23T01:31:41.851 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.845+0000 7f3eb93f2700 1 -- start start 2023-06-23T01:31:41.851 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb93f2700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c4890 0x7f3eb416adc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.851 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb93f2700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb415a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.851 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb93f2700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb415a710 0x7f3eb40692a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.851 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb93f2700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4068e10 con 0x7f3eb415a710 2023-06-23T01:31:41.851 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb93f2700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb416b300 con 0x7f3eb40c4890 2023-06-23T01:31:41.851 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb93f2700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eb4068520 con 0x7f3eb40c93a0 2023-06-23T01:31:41.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb415a710 0x7f3eb40692a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb415a710 0x7f3eb40692a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57744/0 (socket says 172.21.15.92:57744) 2023-06-23T01:31:41.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 -- 172.21.15.92:0/431265491 learned_addr learned my addr 172.21.15.92:0/431265491 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:41.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb3fff700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c4890 0x7f3eb416adc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb37fe700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb415a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 -- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 msgr2=0x7f3eb415a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb415a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.852 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 -- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c4890 msgr2=0x7f3eb416adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:41.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c4890 0x7f3eb416adc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:41.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 -- 172.21.15.92:0/431265491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ea40093c0 con 0x7f3eb415a710 2023-06-23T01:31:41.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb3fff700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c4890 0x7f3eb416adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:31:41.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.846+0000 7f3eb8bf1700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb415a710 0x7f3eb40692a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00cc40 tx=0x7f3eac001a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.847+0000 7f3e9bfff700 1 -- 172.21.15.92:0/431265491 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac00a9c0 con 0x7f3eb415a710 2023-06-23T01:31:41.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.847+0000 7f3e9bfff700 1 -- 172.21.15.92:0/431265491 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eac00ab40 con 0x7f3eb415a710 2023-06-23T01:31:41.853 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.847+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ea8009420 con 0x7f3eb415a710 2023-06-23T01:31:41.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.847+0000 7f3e9bfff700 1 -- 172.21.15.92:0/431265491 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eac0070f0 con 0x7f3eb415a710 2023-06-23T01:31:41.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.847+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eb415af50 con 0x7f3eb415a710 2023-06-23T01:31:41.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.847+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eb40690f0 con 0x7f3eb415a710 2023-06-23T01:31:41.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.848+0000 7f3e9bfff700 1 -- 172.21.15.92:0/431265491 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f3eac0074e0 con 0x7f3eb415a710 2023-06-23T01:31:41.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.848+0000 7f3e9bfff700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f3ea0066e90 0x7f3ea0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:41.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.848+0000 7f3e9bfff700 1 -- 172.21.15.92:0/431265491 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f3eac084610 con 0x7f3eb415a710 2023-06-23T01:31:41.854 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.848+0000 7f3eb3fff700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f3ea0066e90 0x7f3ea0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:41.855 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.849+0000 7f3eb3fff700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f3ea0066e90 0x7f3ea0069340 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009390 tx=0x7f3ea4008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:41.855 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:41.852+0000 7f3e9bfff700 1 -- 172.21.15.92:0/431265491 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f3eac051bb0 con 0x7f3eb415a710 2023-06-23T01:31:42.078 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.077+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3eb4068f50 con 0x7f3eb415a710 2023-06-23T01:31:42.078 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.077+0000 7f3e9bfff700 1 -- 172.21.15.92:0/431265491 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3530 (secure 0 0 0) 0x7f3eb4068f50 con 0x7f3eb415a710 2023-06-23T01:31:42.079 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:31:42.079 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4262,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6835/2533765098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6834","nonce":2533765098},{"type":"v1","addr":"172.21.15.119:6835","nonce":2533765098}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6836/3136947343","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6834","nonce":3136947343},{"type":"v1","addr":"172.21.15.92:6836","nonce":3136947343}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4281,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.92:6837/3505842103","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6835","nonce":3505842103},{"type":"v1","addr":"172.21.15.92:6837","nonce":3505842103}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4309,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.119:6833/4266784946","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6832","nonce":4266784946},{"type":"v1","addr":"172.21.15.119:6833","nonce":4266784946}]},"join_fscid":-1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-06-23T01:31:42.080 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f3ea0066e90 msgr2=0x7f3ea0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.081 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f3ea0066e90 0x7f3ea0069340 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009390 tx=0x7f3ea4008040 comp rx=0 tx=0).stop 2023-06-23T01:31:42.081 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb415a710 msgr2=0x7f3eb40692a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.081 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb415a710 0x7f3eb40692a0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00cc40 tx=0x7f3eac001a50 comp rx=0 tx=0).stop 2023-06-23T01:31:42.081 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 shutdown_connections 2023-06-23T01:31:42.081 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f3ea0066e90 0x7f3ea0069340 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.081 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f3eb415a710 0x7f3eb40692a0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.081 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f3eb40c4890 0x7f3eb416adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.081 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 --2- 172.21.15.92:0/431265491 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f3eb40c93a0 0x7f3eb415a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.082 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 >> 172.21.15.92:0/431265491 conn(0x7f3eb4067bf0 msgr2=0x7f3eb40ce1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:42.083 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 shutdown_connections 2023-06-23T01:31:42.083 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.079+0000 7f3eb93f2700 1 -- 172.21.15.92:0/431265491 wait complete. 2023-06-23T01:31:42.084 INFO:teuthology.orchestra.run.smithi092.stderr:dumped fsmap epoch 3 2023-06-23T01:31:42.093 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:31:42.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.204+0000 7f080a85e700 1 Processor -- start 2023-06-23T01:31:42.206 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.204+0000 7f080a85e700 1 -- start start 2023-06-23T01:31:42.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f080a85e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f08040cbd60 0x7f0804159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f080a85e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 0x7f0804068380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f080a85e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08040ccec0 con 0x7f0804159430 2023-06-23T01:31:42.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f080a85e700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08040cc2b0 con 0x7f08040cbd60 2023-06-23T01:31:42.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f080a85e700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08040cc590 con 0x7f08040b2790 2023-06-23T01:31:42.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f0803fff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f0804159430 0x7f08040cd350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36088/0 (socket says 172.21.15.92:36088) 2023-06-23T01:31:42.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f0803fff700 1 -- 172.21.15.92:0/2309479044 learned_addr learned my addr 172.21.15.92:0/2309479044 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:42.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f08037fe700 1 --2- 172.21.15.92:0/2309479044 >> v2:172.21.15.119:3300/0 conn(0x7f08040cbd60 0x7f0804159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.210 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f0808dfb700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 0x7f0804068380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.205+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120753993 0 0) 0x7f08040ccec0 con 0x7f0804159430 2023-06-23T01:31:42.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07ec003570 con 0x7f0804159430 2023-06-23T01:31:42.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3809238678 0 0) 0x7f07ec003570 con 0x7f0804159430 2023-06-23T01:31:42.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 msgr2=0x7f0804068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 0x7f0804068380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 >> v2:172.21.15.119:3300/0 conn(0x7f08040cbd60 msgr2=0x7f0804159800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 --2- 172.21.15.92:0/2309479044 >> v2:172.21.15.119:3300/0 conn(0x7f08040cbd60 0x7f0804159800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.211 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08040cc870 con 0x7f0804159430 2023-06-23T01:31:42.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 <== mon.0 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3236472324 0 0) 0x7f07f4001470 con 0x7f0804159430 2023-06-23T01:31:42.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f08037fe700 1 --2- 172.21.15.92:0/2309479044 >> v2:172.21.15.119:3300/0 conn(0x7f08040cbd60 0x7f0804159800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:42.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0808dfb700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 0x7f0804068380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:42.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 >> v1:172.21.15.92:6789/0 conn(0x7f0804159430 legacy=0x7f08040cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.212 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec003a60 0x7f07ec003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07ec004810 0x7f07ec004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 0x7f07ec0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08040cc2b0 con 0x7f07ec004810 2023-06-23T01:31:42.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08040cc590 con 0x7f07ec003a60 2023-06-23T01:31:42.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07ec00ba60 con 0x7f08040b2790 2023-06-23T01:31:42.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f08037fe700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07ec004810 0x7f07ec004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.213 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0803fff700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec003a60 0x7f07ec003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f0808dfb700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 0x7f07ec0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f08037fe700 1 -- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 msgr2=0x7f07ec0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f08037fe700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f08040b2790 0x7f07ec0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f08037fe700 1 -- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec003a60 msgr2=0x7f07ec003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f08037fe700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec003a60 0x7f07ec003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f08037fe700 1 -- 172.21.15.92:0/2309479044 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07fc009a20 con 0x7f07ec004810 2023-06-23T01:31:42.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.206+0000 7f08037fe700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07ec004810 0x7f07ec004c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f08040cc1e0 tx=0x7f07f80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aab3a754c6915836 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8009740 con 0x7f07ec004810 2023-06-23T01:31:42.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f800e920 con 0x7f07ec004810 2023-06-23T01:31:42.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f0802ffd700 1 -- 172.21.15.92:0/2309479044 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f800ecc0 con 0x7f07ec004810 2023-06-23T01:31:42.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f080a85e700 1 -- 172.21.15.92:0/2309479044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07ec004810 msgr2=0x7f07ec004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f080a85e700 1 --2- 172.21.15.92:0/2309479044 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f07ec004810 0x7f07ec004c80 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f08040cc1e0 tx=0x7f07f80091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f0808dfb700 1 -- 172.21.15.92:0/2309479044 reap_dead start 2023-06-23T01:31:42.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f080a85e700 1 -- 172.21.15.92:0/2309479044 shutdown_connections 2023-06-23T01:31:42.215 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f080a85e700 1 -- 172.21.15.92:0/2309479044 >> 172.21.15.92:0/2309479044 conn(0x7f0804070290 msgr2=0x7f0804070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:42.216 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f080a85e700 1 -- 172.21.15.92:0/2309479044 shutdown_connections 2023-06-23T01:31:42.216 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.207+0000 7f080a85e700 1 -- 172.21.15.92:0/2309479044 wait complete. 2023-06-23T01:31:42.216 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.208+0000 7f080a85e700 1 Processor -- start 2023-06-23T01:31:42.218 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.208+0000 7f080a85e700 1 -- start start 2023-06-23T01:31:42.218 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f080a85e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec004810 0x7f0804152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.218 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f080a85e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040c65b0 0x7f08040c5690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.219 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f080a85e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f080416a4d0 0x7f08040b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.219 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f080a85e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08041502f0 con 0x7f08040c65b0 2023-06-23T01:31:42.219 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f080a85e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f080416b400 con 0x7f07ec004810 2023-06-23T01:31:42.219 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f080a85e700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08040c5200 con 0x7f080416a4d0 2023-06-23T01:31:42.219 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f08037fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040c65b0 0x7f08040c5690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.219 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f08037fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040c65b0 0x7f08040c5690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57770/0 (socket says 172.21.15.92:57770) 2023-06-23T01:31:42.219 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f08037fe700 1 -- 172.21.15.92:0/3935561804 learned_addr learned my addr 172.21.15.92:0/3935561804 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:42.219 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f0803fff700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec004810 0x7f0804152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.209+0000 7f0808dfb700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f080416a4d0 0x7f08040b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f08037fe700 1 -- 172.21.15.92:0/3935561804 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f080416a4d0 msgr2=0x7f08040b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f08037fe700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f080416a4d0 0x7f08040b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f08037fe700 1 -- 172.21.15.92:0/3935561804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec004810 msgr2=0x7f0804152af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f08037fe700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec004810 0x7f0804152af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f08037fe700 1 -- 172.21.15.92:0/3935561804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0804153030 con 0x7f08040c65b0 2023-06-23T01:31:42.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f08037fe700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040c65b0 0x7f08040c5690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f07f800e790 tx=0x7f07f801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f07ebfff700 1 -- 172.21.15.92:0/3935561804 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f8023070 con 0x7f08040c65b0 2023-06-23T01:31:42.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f07ebfff700 1 -- 172.21.15.92:0/3935561804 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07f800eca0 con 0x7f08040c65b0 2023-06-23T01:31:42.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f080416ae40 con 0x7f08040c65b0 2023-06-23T01:31:42.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f07ebfff700 1 -- 172.21.15.92:0/3935561804 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07f801e260 con 0x7f08040c65b0 2023-06-23T01:31:42.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.210+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08040c4e90 con 0x7f08040c65b0 2023-06-23T01:31:42.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.211+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f080404c1b0 con 0x7f08040c65b0 2023-06-23T01:31:42.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.211+0000 7f07ebfff700 1 -- 172.21.15.92:0/3935561804 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f07f8013680 con 0x7f08040c65b0 2023-06-23T01:31:42.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.212+0000 7f07ebfff700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f07f0066e90 0x7f07f0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.212+0000 7f07ebfff700 1 -- 172.21.15.92:0/3935561804 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f07f8085bc0 con 0x7f08040c65b0 2023-06-23T01:31:42.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.212+0000 7f0803fff700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f07f0066e90 0x7f07f0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.212+0000 7f0803fff700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f07f0066e90 0x7f07f0069340 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07f4001ed0 tx=0x7f08040cce30 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.216+0000 7f07ebfff700 1 -- 172.21.15.92:0/3935561804 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f07f80530e0 con 0x7f08040c65b0 2023-06-23T01:31:42.403 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.401+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f08040c5340 con 0x7f08040c65b0 2023-06-23T01:31:42.403 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.402+0000 7f07ebfff700 1 -- 172.21.15.92:0/3935561804 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+13647 (secure 0 0 0) 0x7f07f8017020 con 0x7f08040c65b0 2023-06-23T01:31:42.404 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:31:42.404 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":19,"fsid":"c94811cf-f7e5-4c57-b505-a983532f8da8","created":"2023-06-23T01:31:20.907484+0000","modified":"2023-06-23T01:31:41.706899+0000","last_up_change":"2023-06-23T01:31:32.355093+0000","last_in_change":"2023-06-23T01:31:23.354162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:31:32.414191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:31:39.216467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4bcf4cdc-7c98-4a4a-bc56-133f22f4d802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":87954},{"type":"v1","addr":"172.21.15.92:6809","nonce":87954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":87954},{"type":"v1","addr":"172.21.15.92:6811","nonce":87954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":87954},{"type":"v1","addr":"172.21.15.92:6815","nonce":87954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":87954},{"type":"v1","addr":"172.21.15.92:6813","nonce":87954}]},"public_addr":"172.21.15.92:6809/87954","cluster_addr":"172.21.15.92:6811/87954","heartbeat_back_addr":"172.21.15.92:6815/87954","heartbeat_front_addr":"172.21.15.92:6813/87954","state":["exists","up"]},{"osd":1,"uuid":"a2e05a19-0c70-4cb4-aacd-3381219435b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":87955},{"type":"v1","addr":"172.21.15.92:6821","nonce":87955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":87955},{"type":"v1","addr":"172.21.15.92:6826","nonce":87955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":87955},{"type":"v1","addr":"172.21.15.92:6831","nonce":87955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":87955},{"type":"v1","addr":"172.21.15.92:6829","nonce":87955}]},"public_addr":"172.21.15.92:6821/87955","cluster_addr":"172.21.15.92:6826/87955","heartbeat_back_addr":"172.21.15.92:6831/87955","heartbeat_front_addr":"172.21.15.92:6829/87955","state":["exists","up"]},{"osd":2,"uuid":"20f85617-68ff-491f-a436-aa1ad1979595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":87951},{"type":"v1","addr":"172.21.15.92:6801","nonce":87951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":87951},{"type":"v1","addr":"172.21.15.92:6803","nonce":87951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":87951},{"type":"v1","addr":"172.21.15.92:6807","nonce":87951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":87951},{"type":"v1","addr":"172.21.15.92:6805","nonce":87951}]},"public_addr":"172.21.15.92:6801/87951","cluster_addr":"172.21.15.92:6803/87951","heartbeat_back_addr":"172.21.15.92:6807/87951","heartbeat_front_addr":"172.21.15.92:6805/87951","state":["exists","up"]},{"osd":3,"uuid":"3e5b05f4-6b77-4612-9720-38ece0b1388b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":87956},{"type":"v1","addr":"172.21.15.92:6817","nonce":87956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":87956},{"type":"v1","addr":"172.21.15.92:6820","nonce":87956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":87956},{"type":"v1","addr":"172.21.15.92:6827","nonce":87956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":87956},{"type":"v1","addr":"172.21.15.92:6824","nonce":87956}]},"public_addr":"172.21.15.92:6817/87956","cluster_addr":"172.21.15.92:6820/87956","heartbeat_back_addr":"172.21.15.92:6827/87956","heartbeat_front_addr":"172.21.15.92:6824/87956","state":["exists","up"]},{"osd":4,"uuid":"4d94d9c9-4288-477f-801f-c66683295335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6801","nonce":88172},{"type":"v1","addr":"172.21.15.119:6802","nonce":88172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":88172},{"type":"v1","addr":"172.21.15.119:6806","nonce":88172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":88172},{"type":"v1","addr":"172.21.15.119:6814","nonce":88172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":88172},{"type":"v1","addr":"172.21.15.119:6810","nonce":88172}]},"public_addr":"172.21.15.119:6802/88172","cluster_addr":"172.21.15.119:6806/88172","heartbeat_back_addr":"172.21.15.119:6814/88172","heartbeat_front_addr":"172.21.15.119:6810/88172","state":["exists","up"]},{"osd":5,"uuid":"dc6b2f59-8af4-42b2-b7f9-bf36714c358f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":88173},{"type":"v1","addr":"172.21.15.119:6803","nonce":88173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6805","nonce":88173},{"type":"v1","addr":"172.21.15.119:6807","nonce":88173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":88173},{"type":"v1","addr":"172.21.15.119:6815","nonce":88173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":88173},{"type":"v1","addr":"172.21.15.119:6811","nonce":88173}]},"public_addr":"172.21.15.119:6803/88173","cluster_addr":"172.21.15.119:6807/88173","heartbeat_back_addr":"172.21.15.119:6815/88173","heartbeat_front_addr":"172.21.15.119:6811/88173","state":["exists","up"]},{"osd":6,"uuid":"2ac0eb61-39bf-4666-8c24-279bf8b9c51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6817","nonce":88202},{"type":"v1","addr":"172.21.15.119:6819","nonce":88202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6821","nonce":88202},{"type":"v1","addr":"172.21.15.119:6823","nonce":88202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6829","nonce":88202},{"type":"v1","addr":"172.21.15.119:6831","nonce":88202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6825","nonce":88202},{"type":"v1","addr":"172.21.15.119:6827","nonce":88202}]},"public_addr":"172.21.15.119:6819/88202","cluster_addr":"172.21.15.119:6823/88202","heartbeat_back_addr":"172.21.15.119:6831/88202","heartbeat_front_addr":"172.21.15.119:6827/88202","state":["exists","up"]},{"osd":7,"uuid":"c4ff77b0-23dc-477d-adaa-d7fd9108f5da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":88203},{"type":"v1","addr":"172.21.15.119:6818","nonce":88203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":88203},{"type":"v1","addr":"172.21.15.119:6822","nonce":88203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":88203},{"type":"v1","addr":"172.21.15.119:6830","nonce":88203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":88203},{"type":"v1","addr":"172.21.15.119:6826","nonce":88203}]},"public_addr":"172.21.15.119:6818/88203","cluster_addr":"172.21.15.119:6822/88203","heartbeat_back_addr":"172.21.15.119:6830/88203","heartbeat_front_addr":"172.21.15.119:6826/88203","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.542554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.495262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.517893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.443537+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.444113+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.472890+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.556249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.573872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:31:42.406 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f07f0066e90 msgr2=0x7f07f0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.406 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f07f0066e90 0x7f07f0069340 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f07f4001ed0 tx=0x7f08040cce30 comp rx=0 tx=0).stop 2023-06-23T01:31:42.406 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040c65b0 msgr2=0x7f08040c5690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.406 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040c65b0 0x7f08040c5690 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f07f800e790 tx=0x7f07f801a000 comp rx=0 tx=0).stop 2023-06-23T01:31:42.406 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 shutdown_connections 2023-06-23T01:31:42.406 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f07f0066e90 0x7f07f0069340 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.407 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f080416a4d0 0x7f08040b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.407 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f07ec004810 0x7f0804152af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.407 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 --2- 172.21.15.92:0/3935561804 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f08040c65b0 0x7f08040c5690 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.407 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.404+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 >> 172.21.15.92:0/3935561804 conn(0x7f0804070290 msgr2=0x7f08040cd250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:42.407 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.405+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 shutdown_connections 2023-06-23T01:31:42.407 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.405+0000 7f080a85e700 1 -- 172.21.15.92:0/3935561804 wait complete. 2023-06-23T01:31:42.418 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-06-23T01:31:32.414191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-06-23T01:31:39.216467+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2023-06-23T01:31:42.419 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-06-23T01:31:42.535 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.534+0000 7fc4f5c45700 1 Processor -- start 2023-06-23T01:31:42.536 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.534+0000 7fc4f5c45700 1 -- start start 2023-06-23T01:31:42.539 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4f5c45700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f0159310 0x7fc4f01596e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.539 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4f5c45700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fc4f00b00e0 0x7fc4f00b0540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.539 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4f5c45700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4f00b0a80 con 0x7fc4f00b23d0 2023-06-23T01:31:42.539 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4f5c45700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4f00b2e10 con 0x7fc4f00b00e0 2023-06-23T01:31:42.539 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4f5c45700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00b15c0 con 0x7fc4f0159310 2023-06-23T01:31:42.539 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4effff700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fc4f00b23d0 0x7fc4f015dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36100/0 (socket says 172.21.15.92:36100) 2023-06-23T01:31:42.540 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4effff700 1 -- 172.21.15.92:0/3002319139 learned_addr learned my addr 172.21.15.92:0/3002319139 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:42.540 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4f5444700 1 --2- 172.21.15.92:0/3002319139 >> v2:172.21.15.119:3300/0 conn(0x7fc4f00b00e0 0x7fc4f00b0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.540 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4f4c43700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f0159310 0x7fc4f01596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.540 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226980821 0 0) 0x7fc4e0001110 con 0x7fc4f00b23d0 2023-06-23T01:31:42.540 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4dc003570 con 0x7fc4f00b23d0 2023-06-23T01:31:42.540 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 90183978 0 0) 0x7fc4dc003570 con 0x7fc4f00b23d0 2023-06-23T01:31:42.540 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f0159310 msgr2=0x7fc4f01596e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f0159310 0x7fc4f01596e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 >> v2:172.21.15.119:3300/0 conn(0x7fc4f00b00e0 msgr2=0x7fc4f00b0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 --2- 172.21.15.92:0/3002319139 >> v2:172.21.15.119:3300/0 conn(0x7fc4f00b00e0 0x7fc4f00b0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f00cb7d0 con 0x7fc4f00b23d0 2023-06-23T01:31:42.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 <== mon.0 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3236472324 0 0) 0x7fc4e0001600 con 0x7fc4f00b23d0 2023-06-23T01:31:42.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4f5444700 1 --2- 172.21.15.92:0/3002319139 >> v2:172.21.15.119:3300/0 conn(0x7fc4f00b00e0 0x7fc4f00b0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:42.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.535+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 >> v1:172.21.15.92:6789/0 conn(0x7fc4f00b23d0 legacy=0x7fc4f015dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.541 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f0159310 0x7fc4dc003a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4dc003fe0 0x7fc4dc004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4dc004d50 0x7fc4dc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00b2e10 con 0x7fc4dc004d50 2023-06-23T01:31:42.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc00ba60 con 0x7fc4dc003fe0 2023-06-23T01:31:42.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc00bbd0 con 0x7fc4f0159310 2023-06-23T01:31:42.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4f4c43700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f0159310 0x7fc4dc003a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4f5444700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4dc004d50 0x7fc4dc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.542 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4effff700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4dc003fe0 0x7fc4dc004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4f5444700 1 -- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f0159310 msgr2=0x7fc4dc003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4f5444700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f0159310 0x7fc4dc003a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4f5444700 1 -- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4dc003fe0 msgr2=0x7fc4dc004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4f5444700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4dc003fe0 0x7fc4dc004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4f5444700 1 -- 172.21.15.92:0/3002319139 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc003570 con 0x7fc4dc004d50 2023-06-23T01:31:42.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4f5444700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4dc004d50 0x7fc4dc0051c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009df0 tx=0x7fc4d80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adbcc59151ad4ff6 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.543 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d800e8c0 con 0x7fc4dc004d50 2023-06-23T01:31:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d800ea20 con 0x7fc4dc004d50 2023-06-23T01:31:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.536+0000 7fc4ef7fe700 1 -- 172.21.15.92:0/3002319139 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d800edc0 con 0x7fc4dc004d50 2023-06-23T01:31:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.537+0000 7fc4f5c45700 1 -- 172.21.15.92:0/3002319139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4dc004d50 msgr2=0x7fc4dc0051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.537+0000 7fc4f5c45700 1 --2- 172.21.15.92:0/3002319139 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4dc004d50 0x7fc4dc0051c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fc4d8009df0 tx=0x7fc4d80091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.537+0000 7fc4f5444700 1 -- 172.21.15.92:0/3002319139 reap_dead start 2023-06-23T01:31:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.537+0000 7fc4f5c45700 1 -- 172.21.15.92:0/3002319139 shutdown_connections 2023-06-23T01:31:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.537+0000 7fc4f5c45700 1 -- 172.21.15.92:0/3002319139 >> 172.21.15.92:0/3002319139 conn(0x7fc4f0070290 msgr2=0x7fc4f0070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:42.544 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.537+0000 7fc4f5c45700 1 -- 172.21.15.92:0/3002319139 shutdown_connections 2023-06-23T01:31:42.545 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.537+0000 7fc4f5c45700 1 -- 172.21.15.92:0/3002319139 wait complete. 2023-06-23T01:31:42.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.538+0000 7fc4f5c45700 1 Processor -- start 2023-06-23T01:31:42.547 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.538+0000 7fc4f5c45700 1 -- start start 2023-06-23T01:31:42.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5c45700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f0156da0 0x7fc4f00cf500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5c45700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4f0157170 0x7fc4f015b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5c45700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f01575e0 0x7fc4f015ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5c45700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00cf070 con 0x7fc4f0157170 2023-06-23T01:31:42.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5c45700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00d08a0 con 0x7fc4f0156da0 2023-06-23T01:31:42.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5c45700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4f00d0b80 con 0x7fc4f01575e0 2023-06-23T01:31:42.548 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4f0157170 0x7fc4f015b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4f0157170 0x7fc4f015b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57794/0 (socket says 172.21.15.92:57794) 2023-06-23T01:31:42.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 -- 172.21.15.92:0/2126946585 learned_addr learned my addr 172.21.15.92:0/2126946585 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:42.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f4c43700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f0156da0 0x7fc4f00cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5444700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f01575e0 0x7fc4f015ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 -- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f01575e0 msgr2=0x7fc4f015ba50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f01575e0 0x7fc4f015ba50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.549 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 -- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f0156da0 msgr2=0x7fc4f00cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f0156da0 0x7fc4f00cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 -- 172.21.15.92:0/2126946585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4f00d0e60 con 0x7fc4f0157170 2023-06-23T01:31:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5444700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f01575e0 0x7fc4f015ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:31:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4effff700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4f0157170 0x7fc4f015b510 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0007b70 tx=0x7fc4e0007a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4ecff9700 1 -- 172.21.15.92:0/2126946585 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e000f8a0 con 0x7fc4f0157170 2023-06-23T01:31:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4ecff9700 1 -- 172.21.15.92:0/2126946585 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4e000fa00 con 0x7fc4f0157170 2023-06-23T01:31:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4f00ced00 con 0x7fc4f0157170 2023-06-23T01:31:42.550 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4ecff9700 1 -- 172.21.15.92:0/2126946585 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4e000fdc0 con 0x7fc4f0157170 2023-06-23T01:31:42.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.539+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4f015d7d0 con 0x7fc4f0157170 2023-06-23T01:31:42.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.540+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4f0067f00 con 0x7fc4f0157170 2023-06-23T01:31:42.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.541+0000 7fc4ecff9700 1 -- 172.21.15.92:0/2126946585 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fc4e0013600 con 0x7fc4f0157170 2023-06-23T01:31:42.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.541+0000 7fc4ecff9700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fc4d0066e90 0x7fc4d0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.541+0000 7fc4ecff9700 1 -- 172.21.15.92:0/2126946585 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fc4e00842e0 con 0x7fc4f0157170 2023-06-23T01:31:42.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.541+0000 7fc4f4c43700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fc4d0066e90 0x7fc4d0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.541+0000 7fc4f4c43700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fc4d0066e90 0x7fc4d0069340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80094d0 tx=0x7fc4e800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.551 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.545+0000 7fc4ecff9700 1 -- 172.21.15.92:0/2126946585 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fc4e0051800 con 0x7fc4f0157170 2023-06-23T01:31:42.731 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.730+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc4f00b0a80 con 0x7fc4f0157170 2023-06-23T01:31:42.731 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.730+0000 7fc4ecff9700 1 -- 172.21.15.92:0/2126946585 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc4f00b0a80 con 0x7fc4f0157170 2023-06-23T01:31:42.732 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 1 2023-06-23T01:31:42.734 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fc4d0066e90 msgr2=0x7fc4d0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.734 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fc4d0066e90 0x7fc4d0069340 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc4e80094d0 tx=0x7fc4e800b040 comp rx=0 tx=0).stop 2023-06-23T01:31:42.734 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4f0157170 msgr2=0x7fc4f015b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.734 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4f0157170 0x7fc4f015b510 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc4e0007b70 tx=0x7fc4e0007a80 comp rx=0 tx=0).stop 2023-06-23T01:31:42.735 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 shutdown_connections 2023-06-23T01:31:42.735 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fc4d0066e90 0x7fc4d0069340 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.735 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fc4f01575e0 0x7fc4f015ba50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.735 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fc4f0156da0 0x7fc4f00cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.735 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 --2- 172.21.15.92:0/2126946585 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fc4f0157170 0x7fc4f015b510 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.735 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 >> 172.21.15.92:0/2126946585 conn(0x7fc4f0070290 msgr2=0x7fc4f00b4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:42.735 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 shutdown_connections 2023-06-23T01:31:42.736 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.732+0000 7fc4f5c45700 1 -- 172.21.15.92:0/2126946585 wait complete. 2023-06-23T01:31:42.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-06-23T01:31:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.860+0000 7f53aa037700 1 Processor -- start 2023-06-23T01:31:42.861 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.860+0000 7f53aa037700 1 -- start start 2023-06-23T01:31:42.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53aa037700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f53a4153e50 0x7f53a40d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53aa037700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f53a40b2010 0x7f53a40b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53aa037700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a40cedd0 con 0x7f53a4153a80 2023-06-23T01:31:42.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53aa037700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a40d14c0 con 0x7f53a4153e50 2023-06-23T01:31:42.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53aa037700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53a40b29d0 con 0x7f53a40b2010 2023-06-23T01:31:42.863 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a2ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f53a4153e50 0x7f53a40d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a2ffd700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f53a4153e50 0x7f53a40d0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57808/0 (socket says 172.21.15.92:57808) 2023-06-23T01:31:42.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a2ffd700 1 -- 172.21.15.92:0/2098850187 learned_addr learned my addr 172.21.15.92:0/2098850187 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:42.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a3fff700 1 --2- 172.21.15.92:0/2098850187 >> v2:172.21.15.119:3300/0 conn(0x7f53a40b2010 0x7f53a40b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a2ffd700 1 -- 172.21.15.92:0/2098850187 >> v2:172.21.15.119:3300/0 conn(0x7f53a40b2010 msgr2=0x7f53a40b2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a2ffd700 1 --2- 172.21.15.92:0/2098850187 >> v2:172.21.15.119:3300/0 conn(0x7f53a40b2010 0x7f53a40b2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a2ffd700 1 -- 172.21.15.92:0/2098850187 >> v1:172.21.15.119:6789/0 conn(0x7f53a4153a80 legacy=0x7f53a415a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:42.864 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a2ffd700 1 -- 172.21.15.92:0/2098850187 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a415aed0 con 0x7f53a4153e50 2023-06-23T01:31:42.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a2ffd700 1 --2- 172.21.15.92:0/2098850187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a4153e50 0x7f53a40d0590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53a40ced40 tx=0x7f5394009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce11860c88d7df9 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a27fc700 1 -- 172.21.15.92:0/2098850187 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539400b820 con 0x7f53a4153e50 2023-06-23T01:31:42.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a27fc700 1 -- 172.21.15.92:0/2098850187 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f539400b980 con 0x7f53a4153e50 2023-06-23T01:31:42.865 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.861+0000 7f53a27fc700 1 -- 172.21.15.92:0/2098850187 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f539400bd20 con 0x7f53a4153e50 2023-06-23T01:31:42.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.862+0000 7f53aa037700 1 -- 172.21.15.92:0/2098850187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a4153e50 msgr2=0x7f53a40d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.862+0000 7f53aa037700 1 --2- 172.21.15.92:0/2098850187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a4153e50 0x7f53a40d0590 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f53a40ced40 tx=0x7f5394009250 comp rx=0 tx=0).stop 2023-06-23T01:31:42.868 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.862+0000 7f53aa037700 1 -- 172.21.15.92:0/2098850187 shutdown_connections 2023-06-23T01:31:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.862+0000 7f53aa037700 1 --2- 172.21.15.92:0/2098850187 >> v2:172.21.15.119:3300/0 conn(0x7f53a40b2010 0x7f53a40b2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.862+0000 7f53aa037700 1 --2- 172.21.15.92:0/2098850187 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a4153e50 0x7f53a40d0590 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.862+0000 7f53aa037700 1 -- 172.21.15.92:0/2098850187 >> 172.21.15.92:0/2098850187 conn(0x7f53a4070290 msgr2=0x7f53a4070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.862+0000 7f53aa037700 1 -- 172.21.15.92:0/2098850187 shutdown_connections 2023-06-23T01:31:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.862+0000 7f53aa037700 1 -- 172.21.15.92:0/2098850187 wait complete. 2023-06-23T01:31:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.863+0000 7f53aa037700 1 Processor -- start 2023-06-23T01:31:42.869 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.863+0000 7f53aa037700 1 -- start start 2023-06-23T01:31:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53aa037700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a40b2010 0x7f53a41428a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53aa037700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f53a4153a80 0x7f53a4159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53aa037700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a4153e50 0x7f53a415a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53aa037700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a4142de0 con 0x7f53a40b2010 2023-06-23T01:31:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53aa037700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a4142410 con 0x7f53a4153e50 2023-06-23T01:31:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53aa037700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53a4141800 con 0x7f53a4153a80 2023-06-23T01:31:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a40b2010 0x7f53a41428a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.870 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a37fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a40b2010 0x7f53a41428a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57818/0 (socket says 172.21.15.92:57818) 2023-06-23T01:31:42.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a37fe700 1 -- 172.21.15.92:0/2692104414 learned_addr learned my addr 172.21.15.92:0/2692104414 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:42.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a2ffd700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f53a4153a80 0x7f53a4159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a3fff700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a4153e50 0x7f53a415a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a37fe700 1 -- 172.21.15.92:0/2692104414 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f53a4153a80 msgr2=0x7f53a4159db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a37fe700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f53a4153a80 0x7f53a4159db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a37fe700 1 -- 172.21.15.92:0/2692104414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a4153e50 msgr2=0x7f53a415a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:42.871 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a37fe700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a4153e50 0x7f53a415a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.864+0000 7f53a37fe700 1 -- 172.21.15.92:0/2692104414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53940094f0 con 0x7f53a40b2010 2023-06-23T01:31:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.865+0000 7f53a37fe700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a40b2010 0x7f53a41428a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f539800c590 tx=0x7f5398007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.865+0000 7f538b7fe700 1 -- 172.21.15.92:0/2692104414 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398011590 con 0x7f53a40b2010 2023-06-23T01:31:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.865+0000 7f538b7fe700 1 -- 172.21.15.92:0/2692104414 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5398011730 con 0x7f53a40b2010 2023-06-23T01:31:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.865+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a4141ae0 con 0x7f53a40b2010 2023-06-23T01:31:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.865+0000 7f538b7fe700 1 -- 172.21.15.92:0/2692104414 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5398011af0 con 0x7f53a40b2010 2023-06-23T01:31:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.865+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a41420a0 con 0x7f53a40b2010 2023-06-23T01:31:42.872 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.866+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a404c1b0 con 0x7f53a40b2010 2023-06-23T01:31:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.866+0000 7f538b7fe700 1 -- 172.21.15.92:0/2692104414 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f5398012420 con 0x7f53a40b2010 2023-06-23T01:31:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.866+0000 7f538b7fe700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5390066e90 0x7f5390069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.866+0000 7f53a2ffd700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5390066e90 0x7f5390069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.866+0000 7f538b7fe700 1 -- 172.21.15.92:0/2692104414 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f5398083580 con 0x7f53a40b2010 2023-06-23T01:31:42.873 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.867+0000 7f53a2ffd700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5390066e90 0x7f5390069340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f53a40ced40 tx=0x7f5394017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:42.874 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:42.870+0000 7f538b7fe700 1 -- 172.21.15.92:0/2692104414 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f539804b590 con 0x7f53a40b2010 2023-06-23T01:31:43.053 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.052+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f53a40b6420 con 0x7f53a40b2010 2023-06-23T01:31:43.054 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.052+0000 7f538b7fe700 1 -- 172.21.15.92:0/2692104414 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f53980170a0 con 0x7f53a40b2010 2023-06-23T01:31:43.054 INFO:teuthology.orchestra.run.smithi092.stdout:pg_num: 8 2023-06-23T01:31:43.056 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5390066e90 msgr2=0x7f5390069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.057 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5390066e90 0x7f5390069340 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f53a40ced40 tx=0x7f5394017000 comp rx=0 tx=0).stop 2023-06-23T01:31:43.057 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a40b2010 msgr2=0x7f53a41428a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.057 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a40b2010 0x7f53a41428a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f539800c590 tx=0x7f5398007f30 comp rx=0 tx=0).stop 2023-06-23T01:31:43.058 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 shutdown_connections 2023-06-23T01:31:43.058 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f5390066e90 0x7f5390069340 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.058 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f53a4153e50 0x7f53a415a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.059 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f53a40b2010 0x7f53a41428a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.059 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 --2- 172.21.15.92:0/2692104414 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f53a4153a80 0x7f53a4159db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.059 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.054+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 >> 172.21.15.92:0/2692104414 conn(0x7f53a4070290 msgr2=0x7f53a4153050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:43.059 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.055+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 shutdown_connections 2023-06-23T01:31:43.059 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.055+0000 7f53aa037700 1 -- 172.21.15.92:0/2692104414 wait complete. 2023-06-23T01:31:43.068 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-06-23T01:31:43.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.179+0000 7fb0201fd700 1 Processor -- start 2023-06-23T01:31:43.181 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.180+0000 7fb0201fd700 1 -- start start 2023-06-23T01:31:43.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.180+0000 7fb0201fd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0180cbd60 0x7fb018159800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.180+0000 7fb0201fd700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0180b2790 0x7fb018068380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.180+0000 7fb0201fd700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0180ccec0 con 0x7fb018159430 2023-06-23T01:31:43.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.180+0000 7fb0201fd700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0180cc2b0 con 0x7fb0180b2790 2023-06-23T01:31:43.184 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.180+0000 7fb0201fd700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180cc590 con 0x7fb0180cbd60 2023-06-23T01:31:43.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01d798700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0180cbd60 0x7fb018159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01e79a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0180b2790 0x7fb018068380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01d798700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0180cbd60 0x7fb018159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3301/0 says I am v2:172.21.15.92:47958/0 (socket says 172.21.15.92:47958) 2023-06-23T01:31:43.185 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01df99700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7fb018159430 0x7fb0180cd350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.92:38080/0 (socket says 172.21.15.92:38080) 2023-06-23T01:31:43.186 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01d798700 1 -- 172.21.15.92:0/3703899162 learned_addr learned my addr 172.21.15.92:0/3703899162 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:43.186 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01e79a700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fb0180b2790 0x7fb018068380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.119:3300/0 says I am v2:172.21.15.92:40786/0 (socket says 172.21.15.92:40786) 2023-06-23T01:31:43.186 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01cf97700 1 -- 172.21.15.92:0/3703899162 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 571440811 0 0) 0x7fb0180ccec0 con 0x7fb018159430 2023-06-23T01:31:43.186 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01cf97700 1 -- 172.21.15.92:0/3703899162 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb004003570 con 0x7fb018159430 2023-06-23T01:31:43.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01d798700 1 -- 172.21.15.92:0/3703899162 >> v2:172.21.15.119:3300/0 conn(0x7fb0180b2790 msgr2=0x7fb018068380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:43.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01d798700 1 --2- 172.21.15.92:0/3703899162 >> v2:172.21.15.119:3300/0 conn(0x7fb0180b2790 0x7fb018068380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01d798700 1 -- 172.21.15.92:0/3703899162 >> v1:172.21.15.119:6789/0 conn(0x7fb018159430 legacy=0x7fb0180cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.181+0000 7fb01d798700 1 -- 172.21.15.92:0/3703899162 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0180cc870 con 0x7fb0180cbd60 2023-06-23T01:31:43.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb01d798700 1 --2- 172.21.15.92:0/3703899162 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0180cbd60 0x7fb018159800 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb0080099b0 tx=0x7fb008009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ce3aff7a4dccecf1 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb01cf97700 1 -- 172.21.15.92:0/3703899162 <== mon.2 v2:172.21.15.119:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800b680 con 0x7fb0180cbd60 2023-06-23T01:31:43.187 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb01cf97700 1 -- 172.21.15.92:0/3703899162 <== mon.2 v2:172.21.15.119:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb00800b7e0 con 0x7fb0180cbd60 2023-06-23T01:31:43.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb01cf97700 1 -- 172.21.15.92:0/3703899162 <== mon.2 v2:172.21.15.119:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb00800bb80 con 0x7fb0180cbd60 2023-06-23T01:31:43.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb0201fd700 1 -- 172.21.15.92:0/3703899162 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0180cbd60 msgr2=0x7fb018159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb0201fd700 1 --2- 172.21.15.92:0/3703899162 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0180cbd60 0x7fb018159800 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb0080099b0 tx=0x7fb008009250 comp rx=0 tx=0).stop 2023-06-23T01:31:43.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb0201fd700 1 -- 172.21.15.92:0/3703899162 shutdown_connections 2023-06-23T01:31:43.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb0201fd700 1 --2- 172.21.15.92:0/3703899162 >> v2:172.21.15.119:3300/0 conn(0x7fb0180b2790 0x7fb018068380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb0201fd700 1 --2- 172.21.15.92:0/3703899162 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb0180cbd60 0x7fb018159800 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.188 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb0201fd700 1 -- 172.21.15.92:0/3703899162 >> 172.21.15.92:0/3703899162 conn(0x7fb018070290 msgr2=0x7fb018070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:43.189 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb0201fd700 1 -- 172.21.15.92:0/3703899162 shutdown_connections 2023-06-23T01:31:43.189 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.182+0000 7fb0201fd700 1 -- 172.21.15.92:0/3703899162 wait complete. 2023-06-23T01:31:43.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.183+0000 7fb0201fd700 1 Processor -- start 2023-06-23T01:31:43.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb0201fd700 1 -- start start 2023-06-23T01:31:43.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb0201fd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0180b2790 0x7fb018148af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb0201fd700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0180cbd60 0x7fb0180cf2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.191 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb0201fd700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb018159430 0x7fb01815cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb0201fd700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180d08a0 con 0x7fb0180b2790 2023-06-23T01:31:43.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb0201fd700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180d0d30 con 0x7fb0180cbd60 2023-06-23T01:31:43.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb0201fd700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0180cfc90 con 0x7fb018159430 2023-06-23T01:31:43.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb01df99700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0180b2790 0x7fb018148af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb01df99700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0180b2790 0x7fb018148af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57830/0 (socket says 172.21.15.92:57830) 2023-06-23T01:31:43.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.184+0000 7fb01df99700 1 -- 172.21.15.92:0/889289314 learned_addr learned my addr 172.21.15.92:0/889289314 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:43.192 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01e79a700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb018159430 0x7fb01815cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01d798700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0180cbd60 0x7fb0180cf2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01df99700 1 -- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb018159430 msgr2=0x7fb01815cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01df99700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb018159430 0x7fb01815cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01df99700 1 -- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0180cbd60 msgr2=0x7fb0180cf2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01df99700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0180cbd60 0x7fb0180cf2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01df99700 1 -- 172.21.15.92:0/889289314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0080094f0 con 0x7fb0180b2790 2023-06-23T01:31:43.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01e79a700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb018159430 0x7fb01815cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:31:43.193 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb01df99700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0180b2790 0x7fb018148af0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb01400cd00 tx=0x7fb014001b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb00dffb700 1 -- 172.21.15.92:0/889289314 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb014005df0 con 0x7fb0180b2790 2023-06-23T01:31:43.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb00dffb700 1 -- 172.21.15.92:0/889289314 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01400a960 con 0x7fb0180b2790 2023-06-23T01:31:43.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb00dffb700 1 -- 172.21.15.92:0/889289314 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01400ad20 con 0x7fb0180b2790 2023-06-23T01:31:43.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0180cff70 con 0x7fb0180b2790 2023-06-23T01:31:43.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.185+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0180d0530 con 0x7fb0180b2790 2023-06-23T01:31:43.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.186+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb01804c1b0 con 0x7fb0180b2790 2023-06-23T01:31:43.194 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.186+0000 7fb00dffb700 1 -- 172.21.15.92:0/889289314 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fb014007300 con 0x7fb0180b2790 2023-06-23T01:31:43.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.187+0000 7fb00dffb700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7faffc066e90 0x7faffc069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.187+0000 7fb00dffb700 1 -- 172.21.15.92:0/889289314 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7fb014084430 con 0x7fb0180b2790 2023-06-23T01:31:43.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.187+0000 7fb01d798700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7faffc066e90 0x7faffc069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.187+0000 7fb01d798700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7faffc066e90 0x7faffc069340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb0080099b0 tx=0x7fb008017000 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.195 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.191+0000 7fb00dffb700 1 -- 172.21.15.92:0/889289314 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fb0140519d0 con 0x7fb0180b2790 2023-06-23T01:31:43.378 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.377+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb01816be70 con 0x7fb0180b2790 2023-06-23T01:31:43.379 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.377+0000 7fb00dffb700 1 -- 172.21.15.92:0/889289314 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7fb014001c70 con 0x7fb0180b2790 2023-06-23T01:31:43.379 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:31:43.379 INFO:teuthology.orchestra.run.smithi092.stdout:[ 2023-06-23T01:31:43.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2023-06-23T01:31:43.380 INFO:teuthology.orchestra.run.smithi092.stdout: "poolnum": 1, 2023-06-23T01:31:43.380 INFO:teuthology.orchestra.run.smithi092.stdout: "poolname": ".mgr" 2023-06-23T01:31:43.380 INFO:teuthology.orchestra.run.smithi092.stdout: }, 2023-06-23T01:31:43.380 INFO:teuthology.orchestra.run.smithi092.stdout: { 2023-06-23T01:31:43.380 INFO:teuthology.orchestra.run.smithi092.stdout: "poolnum": 2, 2023-06-23T01:31:43.380 INFO:teuthology.orchestra.run.smithi092.stdout: "poolname": "rbd" 2023-06-23T01:31:43.381 INFO:teuthology.orchestra.run.smithi092.stdout: } 2023-06-23T01:31:43.381 INFO:teuthology.orchestra.run.smithi092.stdout:] 2023-06-23T01:31:43.382 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7faffc066e90 msgr2=0x7faffc069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.382 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7faffc066e90 0x7faffc069340 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb0080099b0 tx=0x7fb008017000 comp rx=0 tx=0).stop 2023-06-23T01:31:43.382 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0180b2790 msgr2=0x7fb018148af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.383 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0180b2790 0x7fb018148af0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb01400cd00 tx=0x7fb014001b30 comp rx=0 tx=0).stop 2023-06-23T01:31:43.383 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 shutdown_connections 2023-06-23T01:31:43.383 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7faffc066e90 0x7faffc069340 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.383 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fb018159430 0x7fb01815cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fb0180b2790 0x7fb018148af0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 --2- 172.21.15.92:0/889289314 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fb0180cbd60 0x7fb0180cf2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 >> 172.21.15.92:0/889289314 conn(0x7fb018070290 msgr2=0x7fb0180b5b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:43.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 shutdown_connections 2023-06-23T01:31:43.385 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.379+0000 7fb0201fd700 1 -- 172.21.15.92:0/889289314 wait complete. 2023-06-23T01:31:43.392 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-06-23T01:31:43.393 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-06-23T01:31:43.510 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.509+0000 7f1478612700 1 Processor -- start 2023-06-23T01:31:43.510 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.509+0000 7f1478612700 1 -- start start 2023-06-23T01:31:43.512 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f1478612700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f14700af520 0x7f14700c2070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.512 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f1478612700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700c25b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.512 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f1478612700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14700b0b90 con 0x7f14700af8f0 2023-06-23T01:31:43.512 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f1478612700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1470156bc0 con 0x7f14700af520 2023-06-23T01:31:43.512 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f1478612700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14700cb210 con 0x7f14700b07a0 2023-06-23T01:31:43.513 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f14763ae700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f14700af520 0x7f14700c2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.513 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f1475bad700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f14700af8f0 0x7f14700b0090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36116/0 (socket says 172.21.15.92:36116) 2023-06-23T01:31:43.513 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f1475bad700 1 -- 172.21.15.92:0/1789069049 learned_addr learned my addr 172.21.15.92:0/1789069049 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:43.513 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f1476baf700 1 --2- 172.21.15.92:0/1789069049 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700c25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.513 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f14763ae700 1 -- 172.21.15.92:0/1789069049 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 msgr2=0x7f14700c25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:43.513 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f14763ae700 1 --2- 172.21.15.92:0/1789069049 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700c25b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.513 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f14763ae700 1 -- 172.21.15.92:0/1789069049 >> v1:172.21.15.92:6789/0 conn(0x7f14700af8f0 legacy=0x7f14700b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.514 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f14763ae700 1 -- 172.21.15.92:0/1789069049 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14700b0dc0 con 0x7f14700af520 2023-06-23T01:31:43.514 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f14753ac700 1 -- 172.21.15.92:0/1789069049 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936447059 0 0) 0x7f14700b0b90 con 0x7f14700af8f0 2023-06-23T01:31:43.514 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.510+0000 7f14763ae700 1 --2- 172.21.15.92:0/1789069049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f14700c2070 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f146c0098e0 tx=0x7f146c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db51edec528702e server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.514 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f14753ac700 1 -- 172.21.15.92:0/1789069049 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c009e80 con 0x7f14700af520 2023-06-23T01:31:43.514 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f14753ac700 1 -- 172.21.15.92:0/1789069049 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c00b860 con 0x7f14700af520 2023-06-23T01:31:43.514 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f14753ac700 1 -- 172.21.15.92:0/1789069049 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c00bbe0 con 0x7f14700af520 2023-06-23T01:31:43.517 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f1478612700 1 -- 172.21.15.92:0/1789069049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 msgr2=0x7f14700c2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.517 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f1478612700 1 --2- 172.21.15.92:0/1789069049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f14700c2070 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f146c0098e0 tx=0x7f146c0091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.517 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f1478612700 1 -- 172.21.15.92:0/1789069049 shutdown_connections 2023-06-23T01:31:43.517 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f1478612700 1 --2- 172.21.15.92:0/1789069049 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700c25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.517 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f1478612700 1 --2- 172.21.15.92:0/1789069049 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f14700c2070 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.518 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f1478612700 1 -- 172.21.15.92:0/1789069049 >> 172.21.15.92:0/1789069049 conn(0x7f1470068520 msgr2=0x7f1470068920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:43.518 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f1478612700 1 -- 172.21.15.92:0/1789069049 shutdown_connections 2023-06-23T01:31:43.518 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.511+0000 7f1478612700 1 -- 172.21.15.92:0/1789069049 wait complete. 2023-06-23T01:31:43.518 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.512+0000 7f1478612700 1 Processor -- start 2023-06-23T01:31:43.518 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.512+0000 7f1478612700 1 -- start start 2023-06-23T01:31:43.518 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1478612700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f1470150040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.518 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1478612700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14700af8f0 0x7f14700c9360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.518 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1478612700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700b1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1478612700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470166ec0 con 0x7f14700af520 2023-06-23T01:31:43.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1478612700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14701671a0 con 0x7f14700af8f0 2023-06-23T01:31:43.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1478612700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1470167480 con 0x7f14700b07a0 2023-06-23T01:31:43.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f14763ae700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f1470150040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f14763ae700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f1470150040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57858/0 (socket says 172.21.15.92:57858) 2023-06-23T01:31:43.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f14763ae700 1 -- 172.21.15.92:0/4100769137 learned_addr learned my addr 172.21.15.92:0/4100769137 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:43.519 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1475bad700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14700af8f0 0x7f14700c9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1476baf700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700b1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f14763ae700 1 -- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 msgr2=0x7f14700b1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f14763ae700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700b1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f14763ae700 1 -- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14700af8f0 msgr2=0x7f14700c9360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f14763ae700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14700af8f0 0x7f14700c9360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f14763ae700 1 -- 172.21.15.92:0/4100769137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c009420 con 0x7f14700af520 2023-06-23T01:31:43.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1475bad700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14700af8f0 0x7f14700c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:31:43.520 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.513+0000 7f1476baf700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:31:43.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.514+0000 7f14763ae700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f1470150040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f147006dc80 tx=0x7f146c017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.514+0000 7f14667fc700 1 -- 172.21.15.92:0/4100769137 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c00bbe0 con 0x7f14700af520 2023-06-23T01:31:43.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.514+0000 7f14667fc700 1 -- 172.21.15.92:0/4100769137 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f146c011420 con 0x7f14700af520 2023-06-23T01:31:43.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.514+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1470167760 con 0x7f14700af520 2023-06-23T01:31:43.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.514+0000 7f14667fc700 1 -- 172.21.15.92:0/4100769137 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f146c0117e0 con 0x7f14700af520 2023-06-23T01:31:43.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.514+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1470150580 con 0x7f14700af520 2023-06-23T01:31:43.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.515+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1470167000 con 0x7f14700af520 2023-06-23T01:31:43.521 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.515+0000 7f14667fc700 1 -- 172.21.15.92:0/4100769137 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f146c011bd0 con 0x7f14700af520 2023-06-23T01:31:43.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.515+0000 7f14667fc700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1454066e90 0x7f1454069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.516+0000 7f14667fc700 1 -- 172.21.15.92:0/4100769137 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5206+0+0 (secure 0 0 0) 0x7f146c085e80 con 0x7f14700af520 2023-06-23T01:31:43.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.516+0000 7f1475bad700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1454066e90 0x7f1454069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.516+0000 7f1475bad700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1454066e90 0x7f1454069340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1460007fd0 tx=0x7f1460009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.522 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.520+0000 7f14667fc700 1 -- 172.21.15.92:0/4100769137 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f1470167000 con 0x7f14700af520 2023-06-23T01:31:43.703 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.702+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f1470167940 con 0x7f14700af520 2023-06-23T01:31:43.825 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.824+0000 7f14667fc700 1 -- 172.21.15.92:0/4100769137 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f146c053300 con 0x7f14700af520 2023-06-23T01:31:43.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1454066e90 msgr2=0x7f1454069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1454066e90 0x7f1454069340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1460007fd0 tx=0x7f1460009040 comp rx=0 tx=0).stop 2023-06-23T01:31:43.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 msgr2=0x7f1470150040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f1470150040 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f147006dc80 tx=0x7f146c017000 comp rx=0 tx=0).stop 2023-06-23T01:31:43.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 shutdown_connections 2023-06-23T01:31:43.830 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1454066e90 0x7f1454069340 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.831 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f14700b07a0 0x7f14700b1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.831 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f14700af520 0x7f1470150040 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.831 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 --2- 172.21.15.92:0/4100769137 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f14700af8f0 0x7f14700c9360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.831 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 >> 172.21.15.92:0/4100769137 conn(0x7f1470068520 msgr2=0x7f14700b0070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:43.831 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 shutdown_connections 2023-06-23T01:31:43.831 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.828+0000 7f1478612700 1 -- 172.21.15.92:0/4100769137 wait complete. 2023-06-23T01:31:43.832 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'cephfs_metadata' created 2023-06-23T01:31:43.842 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-06-23T01:31:43.962 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.961+0000 7f2e13ec1700 1 Processor -- start 2023-06-23T01:31:43.963 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.961+0000 7f2e13ec1700 1 -- start start 2023-06-23T01:31:43.963 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.962+0000 7f2e13ec1700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153a80 0x7f2e0c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.963 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.962+0000 7f2e13ec1700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2e0c0b2010 0x7f2e0c0b2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.964 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.962+0000 7f2e13ec1700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e0c0cedd0 con 0x7f2e0c153e50 2023-06-23T01:31:43.964 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.962+0000 7f2e13ec1700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e0c0d14c0 con 0x7f2e0c0b2010 2023-06-23T01:31:43.964 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.962+0000 7f2e13ec1700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c0b29d0 con 0x7f2e0c153a80 2023-06-23T01:31:43.965 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e11c5d700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153a80 0x7f2e0c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.966 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e1245e700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f2e0c0b2010 0x7f2e0c0b2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.966 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e1145c700 1 --1- >> v1:172.21.15.119:6789/0 conn(0x7f2e0c153e50 0x7f2e0c15a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.119:6789/0 says I am v1:172.21.15.92:38094/0 (socket says 172.21.15.92:38094) 2023-06-23T01:31:43.966 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e1145c700 1 -- 172.21.15.92:0/2654535144 learned_addr learned my addr 172.21.15.92:0/2654535144 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:43.966 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e10c5b700 1 -- 172.21.15.92:0/2654535144 <== mon.1 v1:172.21.15.119:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294514904 0 0) 0x7f2e0c0cedd0 con 0x7f2e0c153e50 2023-06-23T01:31:43.966 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e10c5b700 1 -- 172.21.15.92:0/2654535144 --> v1:172.21.15.119:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2df8003570 con 0x7f2e0c153e50 2023-06-23T01:31:43.966 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e1245e700 1 -- 172.21.15.92:0/2654535144 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153a80 msgr2=0x7f2e0c0d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:43.966 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e1245e700 1 --2- 172.21.15.92:0/2654535144 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153a80 0x7f2e0c0d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.967 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e1245e700 1 -- 172.21.15.92:0/2654535144 >> v1:172.21.15.119:6789/0 conn(0x7f2e0c153e50 legacy=0x7f2e0c15a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.967 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e1245e700 1 -- 172.21.15.92:0/2654535144 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e0c15aed0 con 0x7f2e0c0b2010 2023-06-23T01:31:43.967 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e11c5d700 1 --2- 172.21.15.92:0/2654535144 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153a80 0x7f2e0c0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-06-23T01:31:43.967 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e1245e700 1 --2- 172.21.15.92:0/2654535144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 0x7f2e0c0b2490 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2e040098e0 tx=0x7f2e040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38767c29f3ca1f5b server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.967 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e10c5b700 1 -- 172.21.15.92:0/2654535144 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e04011630 con 0x7f2e0c0b2010 2023-06-23T01:31:43.967 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e10c5b700 1 -- 172.21.15.92:0/2654535144 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e04011790 con 0x7f2e0c0b2010 2023-06-23T01:31:43.967 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.963+0000 7f2e10c5b700 1 -- 172.21.15.92:0/2654535144 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e04011b30 con 0x7f2e0c0b2010 2023-06-23T01:31:43.967 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.964+0000 7f2e13ec1700 1 -- 172.21.15.92:0/2654535144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 msgr2=0x7f2e0c0b2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.968 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.964+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/2654535144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 0x7f2e0c0b2490 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2e040098e0 tx=0x7f2e040091a0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.968 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.964+0000 7f2e13ec1700 1 -- 172.21.15.92:0/2654535144 shutdown_connections 2023-06-23T01:31:43.968 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.964+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/2654535144 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 0x7f2e0c0b2490 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.968 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.964+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/2654535144 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153a80 0x7f2e0c0d0590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.968 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.964+0000 7f2e13ec1700 1 -- 172.21.15.92:0/2654535144 >> 172.21.15.92:0/2654535144 conn(0x7f2e0c070290 msgr2=0x7f2e0c070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:43.968 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.964+0000 7f2e13ec1700 1 -- 172.21.15.92:0/2654535144 shutdown_connections 2023-06-23T01:31:43.968 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.964+0000 7f2e13ec1700 1 -- 172.21.15.92:0/2654535144 wait complete. 2023-06-23T01:31:43.970 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.965+0000 7f2e13ec1700 1 Processor -- start 2023-06-23T01:31:43.971 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.965+0000 7f2e13ec1700 1 -- start start 2023-06-23T01:31:43.971 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e13ec1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 0x7f2e0c15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.971 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e13ec1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0c153a80 0x7f2e0c151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.971 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e13ec1700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153e50 0x7f2e0c151a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.971 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e13ec1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c155990 con 0x7f2e0c153a80 2023-06-23T01:31:43.971 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e13ec1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c154dc0 con 0x7f2e0c0b2010 2023-06-23T01:31:43.971 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e13ec1700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e0c155060 con 0x7f2e0c153e50 2023-06-23T01:31:43.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0c153a80 0x7f2e0c151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0c153a80 0x7f2e0c151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57868/0 (socket says 172.21.15.92:57868) 2023-06-23T01:31:43.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 -- 172.21.15.92:0/1011296592 learned_addr learned my addr 172.21.15.92:0/1011296592 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:43.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e11c5d700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 0x7f2e0c15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1245e700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153e50 0x7f2e0c151a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 -- 172.21.15.92:0/1011296592 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153e50 msgr2=0x7f2e0c151a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153e50 0x7f2e0c151a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.972 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 -- 172.21.15.92:0/1011296592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 msgr2=0x7f2e0c15a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:43.973 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 0x7f2e0c15a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:43.973 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 -- 172.21.15.92:0/1011296592 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08009a20 con 0x7f2e0c153a80 2023-06-23T01:31:43.973 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.966+0000 7f2e1145c700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0c153a80 0x7f2e0c151f90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00c7c0 tx=0x7f2dfc007e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.973 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.967+0000 7f2e01ffb700 1 -- 172.21.15.92:0/1011296592 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc005cd0 con 0x7f2e0c153a80 2023-06-23T01:31:43.973 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.967+0000 7f2e01ffb700 1 -- 172.21.15.92:0/1011296592 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dfc005e30 con 0x7f2e0c153a80 2023-06-23T01:31:43.973 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.967+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e04009420 con 0x7f2e0c153a80 2023-06-23T01:31:43.973 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.967+0000 7f2e01ffb700 1 -- 172.21.15.92:0/1011296592 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2dfc007320 con 0x7f2e0c153a80 2023-06-23T01:31:43.974 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.967+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e0c155620 con 0x7f2e0c153a80 2023-06-23T01:31:43.974 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.968+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e0c04c1b0 con 0x7f2e0c153a80 2023-06-23T01:31:43.974 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.968+0000 7f2e01ffb700 1 -- 172.21.15.92:0/1011296592 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f2dfc01b3b0 con 0x7f2e0c153a80 2023-06-23T01:31:43.974 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.969+0000 7f2e01ffb700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2df0066f10 0x7f2df00693c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:43.974 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.969+0000 7f2e01ffb700 1 -- 172.21.15.92:0/1011296592 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5576+0+0 (secure 0 0 0) 0x7f2dfc083af0 con 0x7f2e0c153a80 2023-06-23T01:31:43.974 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.969+0000 7f2e11c5d700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2df0066f10 0x7f2df00693c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:43.974 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.969+0000 7f2e11c5d700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2df0066f10 0x7f2df00693c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c15ac40 tx=0x7f2e0800b040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:43.975 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:43.973+0000 7f2e01ffb700 1 -- 172.21.15.92:0/1011296592 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f2dfc050e00 con 0x7f2e0c153a80 2023-06-23T01:31:44.156 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.155+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2e0c0b6420 con 0x7f2e0c153a80 2023-06-23T01:31:44.832 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.831+0000 7f2e01ffb700 1 -- 172.21.15.92:0/1011296592 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v21) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2dfc0170d0 con 0x7f2e0c153a80 2023-06-23T01:31:44.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.836+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2df0066f10 msgr2=0x7f2df00693c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:44.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.836+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2df0066f10 0x7f2df00693c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c15ac40 tx=0x7f2e0800b040 comp rx=0 tx=0).stop 2023-06-23T01:31:44.838 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0c153a80 msgr2=0x7f2e0c151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:44.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0c153a80 0x7f2e0c151f90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f2dfc00c7c0 tx=0x7f2dfc007e00 comp rx=0 tx=0).stop 2023-06-23T01:31:44.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 shutdown_connections 2023-06-23T01:31:44.839 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2df0066f10 0x7f2df00693c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f2e0c153e50 0x7f2e0c151a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f2e0c0b2010 0x7f2e0c15a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 --2- 172.21.15.92:0/1011296592 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2e0c153a80 0x7f2e0c151f90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 >> 172.21.15.92:0/1011296592 conn(0x7f2e0c070290 msgr2=0x7f2e0c15c0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:44.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 shutdown_connections 2023-06-23T01:31:44.840 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.837+0000 7f2e13ec1700 1 -- 172.21.15.92:0/1011296592 wait complete. 2023-06-23T01:31:44.841 INFO:teuthology.orchestra.run.smithi092.stderr:pool 'cephfs_data' created 2023-06-23T01:31:44.857 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-06-23T01:31:44.974 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.973+0000 7f1f03505700 1 Processor -- start 2023-06-23T01:31:44.975 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.973+0000 7f1f03505700 1 -- start start 2023-06-23T01:31:44.977 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f03505700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1efc159750 0x7f1efc0a9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:44.977 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f03505700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f1efc159b20 0x7f1efc0c7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:44.977 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f03505700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc0c7640 0x7f1efc06d940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:44.977 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f03505700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1efc0c7ac0 con 0x7f1efc159750 2023-06-23T01:31:44.978 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f03505700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1efc067d40 con 0x7f1efc159b20 2023-06-23T01:31:44.978 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f03505700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0ab9c0 con 0x7f1efc0c7640 2023-06-23T01:31:44.978 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f012a1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1efc159750 0x7f1efc0a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:44.978 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f012a1700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f1efc159750 0x7f1efc0a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57876/0 (socket says 172.21.15.92:57876) 2023-06-23T01:31:44.978 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f012a1700 1 -- 172.21.15.92:0/3711119473 learned_addr learned my addr 172.21.15.92:0/3711119473 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:44.978 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f00aa0700 1 --2- 172.21.15.92:0/3711119473 >> v2:172.21.15.119:3300/0 conn(0x7f1efc159b20 0x7f1efc0c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:44.979 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f01aa2700 1 --2- 172.21.15.92:0/3711119473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc0c7640 0x7f1efc06d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:44.979 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f012a1700 1 -- 172.21.15.92:0/3711119473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc0c7640 msgr2=0x7f1efc06d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:44.979 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f012a1700 1 --2- 172.21.15.92:0/3711119473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc0c7640 0x7f1efc06d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.979 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f012a1700 1 -- 172.21.15.92:0/3711119473 >> v2:172.21.15.119:3300/0 conn(0x7f1efc159b20 msgr2=0x7f1efc0c7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:44.979 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f012a1700 1 --2- 172.21.15.92:0/3711119473 >> v2:172.21.15.119:3300/0 conn(0x7f1efc159b20 0x7f1efc0c7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.979 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.974+0000 7f1f012a1700 1 -- 172.21.15.92:0/3711119473 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1efc06e600 con 0x7f1efc159750 2023-06-23T01:31:44.979 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f012a1700 1 --2- 172.21.15.92:0/3711119473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc159750 0x7f1efc0a9840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0c0590 tx=0x7f1ef8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dac9b9ce786b279 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:44.980 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1ef3fff700 1 -- 172.21.15.92:0/3711119473 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef800b8f0 con 0x7f1efc159750 2023-06-23T01:31:44.980 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1ef3fff700 1 -- 172.21.15.92:0/3711119473 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef800ba50 con 0x7f1efc159750 2023-06-23T01:31:44.980 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1ef3fff700 1 -- 172.21.15.92:0/3711119473 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef800bdf0 con 0x7f1efc159750 2023-06-23T01:31:44.980 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 -- 172.21.15.92:0/3711119473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc159750 msgr2=0x7f1efc0a9840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:44.980 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 --2- 172.21.15.92:0/3711119473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc159750 0x7f1efc0a9840 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0c0590 tx=0x7f1ef8009250 comp rx=0 tx=0).stop 2023-06-23T01:31:44.980 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 -- 172.21.15.92:0/3711119473 shutdown_connections 2023-06-23T01:31:44.980 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 --2- 172.21.15.92:0/3711119473 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc0c7640 0x7f1efc06d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.981 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 --2- 172.21.15.92:0/3711119473 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc159750 0x7f1efc0a9840 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.981 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 --2- 172.21.15.92:0/3711119473 >> v2:172.21.15.119:3300/0 conn(0x7f1efc159b20 0x7f1efc0c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.981 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 -- 172.21.15.92:0/3711119473 >> 172.21.15.92:0/3711119473 conn(0x7f1efc0cd550 msgr2=0x7f1efc0c9530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:44.981 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 -- 172.21.15.92:0/3711119473 shutdown_connections 2023-06-23T01:31:44.981 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.975+0000 7f1f03505700 1 -- 172.21.15.92:0/3711119473 wait complete. 2023-06-23T01:31:44.983 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.976+0000 7f1f03505700 1 Processor -- start 2023-06-23T01:31:44.983 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.976+0000 7f1f03505700 1 -- start start 2023-06-23T01:31:44.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f03505700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc0c7640 0x7f1efc158e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:44.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f03505700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc159750 0x7f1efc0adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:44.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f03505700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc159b20 0x7f1efc0ae320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:44.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f03505700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0ad480 con 0x7f1efc0c7640 2023-06-23T01:31:44.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f03505700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0ac870 con 0x7f1efc159750 2023-06-23T01:31:44.984 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f03505700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1efc0acb50 con 0x7f1efc159b20 2023-06-23T01:31:44.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f012a1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc0c7640 0x7f1efc158e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:44.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f012a1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc0c7640 0x7f1efc158e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57878/0 (socket says 172.21.15.92:57878) 2023-06-23T01:31:44.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.977+0000 7f1f012a1700 1 -- 172.21.15.92:0/23067763 learned_addr learned my addr 172.21.15.92:0/23067763 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:44.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f01aa2700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc159b20 0x7f1efc0ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:44.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f012a1700 1 -- 172.21.15.92:0/23067763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc159b20 msgr2=0x7f1efc0ae320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:44.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f012a1700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc159b20 0x7f1efc0ae320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.985 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f012a1700 1 -- 172.21.15.92:0/23067763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc159750 msgr2=0x7f1efc0adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:44.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f012a1700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc159750 0x7f1efc0adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:44.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f012a1700 1 -- 172.21.15.92:0/23067763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef80094f0 con 0x7f1efc0c7640 2023-06-23T01:31:44.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f01aa2700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc159b20 0x7f1efc0ae320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:31:44.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f012a1700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc0c7640 0x7f1efc158e90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0696c0 tx=0x7f1ef8017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:44.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1ef17fa700 1 -- 172.21.15.92:0/23067763 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef8011e60 con 0x7f1efc0c7640 2023-06-23T01:31:44.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1ef17fa700 1 -- 172.21.15.92:0/23067763 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef801a630 con 0x7f1efc0c7640 2023-06-23T01:31:44.986 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1ef17fa700 1 -- 172.21.15.92:0/23067763 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef801a9f0 con 0x7f1efc0c7640 2023-06-23T01:31:44.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1efc0ace30 con 0x7f1efc0c7640 2023-06-23T01:31:44.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.978+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1efc1593d0 con 0x7f1efc0c7640 2023-06-23T01:31:44.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.979+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1efc04c1b0 con 0x7f1efc0c7640 2023-06-23T01:31:44.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.979+0000 7f1ef17fa700 1 -- 172.21.15.92:0/23067763 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f1ef80233e0 con 0x7f1efc0c7640 2023-06-23T01:31:44.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.980+0000 7f1ef17fa700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1ee0066e90 0x7f1ee0069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:44.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.980+0000 7f1ef17fa700 1 -- 172.21.15.92:0/23067763 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5958+0+0 (secure 0 0 0) 0x7f1ef8085d30 con 0x7f1efc0c7640 2023-06-23T01:31:44.987 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.980+0000 7f1f00aa0700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1ee0066e90 0x7f1ee0069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:44.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.980+0000 7f1f00aa0700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1ee0066e90 0x7f1ee0069340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1eec006fd0 tx=0x7f1eec008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:44.988 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:44.984+0000 7f1ef17fa700 1 -- 172.21.15.92:0/23067763 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f1ef8052ec0 con 0x7f1efc0c7640 2023-06-23T01:31:45.209 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.207+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1efc0c0d60 con 0x7f1efc0c7640 2023-06-23T01:31:45.214 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.213+0000 7f1ef17fa700 1 -- 172.21.15.92:0/23067763 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f1ef80564e0 con 0x7f1efc0c7640 2023-06-23T01:31:45.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.217+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1ee0066e90 msgr2=0x7f1ee0069340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.220 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.217+0000 7f1f03505700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1ee0066e90 0x7f1ee0069340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1eec006fd0 tx=0x7f1eec008040 comp rx=0 tx=0).stop 2023-06-23T01:31:45.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.217+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc0c7640 msgr2=0x7f1efc158e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.217+0000 7f1f03505700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc0c7640 0x7f1efc158e90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1efc0696c0 tx=0x7f1ef8017000 comp rx=0 tx=0).stop 2023-06-23T01:31:45.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.218+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 shutdown_connections 2023-06-23T01:31:45.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.218+0000 7f1f03505700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f1ee0066e90 0x7f1ee0069340 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1eec006fd0 tx=0x7f1eec008040 comp rx=0 tx=0).stop 2023-06-23T01:31:45.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.218+0000 7f1f03505700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f1efc159b20 0x7f1efc0ae320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.218+0000 7f1f03505700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f1efc0c7640 0x7f1efc158e90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.218+0000 7f1f03505700 1 --2- 172.21.15.92:0/23067763 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f1efc159750 0x7f1efc0adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.221 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.218+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 >> 172.21.15.92:0/23067763 conn(0x7f1efc0cd550 msgr2=0x7f1efc069950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:45.222 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.220+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 shutdown_connections 2023-06-23T01:31:45.223 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.222+0000 7f1f03505700 1 -- 172.21.15.92:0/23067763 wait complete. 2023-06-23T01:31:45.236 INFO:teuthology.orchestra.run.smithi092.stderr:new fs with metadata pool 3 and data pool 4 2023-06-23T01:31:45.245 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:31:45.252 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:31:45.290 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:31:45.362 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.361+0000 7f278d971700 1 Processor -- start 2023-06-23T01:31:45.363 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.361+0000 7f278d971700 1 -- start start 2023-06-23T01:31:45.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f278d971700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7f27880af200 0x7f2788069f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f278d971700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f278806a490 0x7f27880ca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f278d971700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27880b1660 con 0x7f2788156230 2023-06-23T01:31:45.364 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f278d971700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27880b2200 con 0x7f278806a490 2023-06-23T01:31:45.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f278d971700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f278806a8a0 con 0x7f27880af200 2023-06-23T01:31:45.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f27877fe700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f278806a490 0x7f27880ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f2786ffd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f2788156230 0x7f27880b1af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36132/0 (socket says 172.21.15.92:36132) 2023-06-23T01:31:45.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f2786ffd700 1 -- 172.21.15.92:0/1771487767 learned_addr learned my addr 172.21.15.92:0/1771487767 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:45.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f27877fe700 1 -- 172.21.15.92:0/1771487767 >> v2:172.21.15.119:3300/0 conn(0x7f27880af200 msgr2=0x7f2788069f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:45.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f277e7fc700 1 --2- 172.21.15.92:0/1771487767 >> v2:172.21.15.119:3300/0 conn(0x7f27880af200 0x7f2788069f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.365 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f27877fe700 1 --2- 172.21.15.92:0/1771487767 >> v2:172.21.15.119:3300/0 conn(0x7f27880af200 0x7f2788069f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f27867fc700 1 -- 172.21.15.92:0/1771487767 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839643407 0 0) 0x7f27880b1660 con 0x7f2788156230 2023-06-23T01:31:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f27877fe700 1 -- 172.21.15.92:0/1771487767 >> v1:172.21.15.92:6789/0 conn(0x7f2788156230 legacy=0x7f27880b1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.362+0000 7f27877fe700 1 -- 172.21.15.92:0/1771487767 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27880b0a90 con 0x7f278806a490 2023-06-23T01:31:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f27877fe700 1 --2- 172.21.15.92:0/1771487767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f278806a490 0x7f27880ca050 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f27880cc9f0 tx=0x7f2780009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0294ee80f5448f3 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f27867fc700 1 -- 172.21.15.92:0/1771487767 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f278000e070 con 0x7f278806a490 2023-06-23T01:31:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f27867fc700 1 -- 172.21.15.92:0/1771487767 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f278000bc30 con 0x7f278806a490 2023-06-23T01:31:45.366 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f27867fc700 1 -- 172.21.15.92:0/1771487767 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2780012430 con 0x7f278806a490 2023-06-23T01:31:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f278d971700 1 -- 172.21.15.92:0/1771487767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f278806a490 msgr2=0x7f27880ca050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f278d971700 1 --2- 172.21.15.92:0/1771487767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f278806a490 0x7f27880ca050 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f27880cc9f0 tx=0x7f2780009780 comp rx=0 tx=0).stop 2023-06-23T01:31:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f278d971700 1 -- 172.21.15.92:0/1771487767 shutdown_connections 2023-06-23T01:31:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f278d971700 1 --2- 172.21.15.92:0/1771487767 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f278806a490 0x7f27880ca050 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f278d971700 1 --2- 172.21.15.92:0/1771487767 >> v2:172.21.15.119:3300/0 conn(0x7f27880af200 0x7f2788069f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.369 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f278d971700 1 -- 172.21.15.92:0/1771487767 >> 172.21.15.92:0/1771487767 conn(0x7f2788068fa0 msgr2=0x7f27880aeaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:45.370 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f278d971700 1 -- 172.21.15.92:0/1771487767 shutdown_connections 2023-06-23T01:31:45.370 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.363+0000 7f278d971700 1 -- 172.21.15.92:0/1771487767 wait complete. 2023-06-23T01:31:45.370 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f278d971700 1 Processor -- start 2023-06-23T01:31:45.370 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f278d971700 1 -- start start 2023-06-23T01:31:45.370 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f278d971700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f278806a490 0x7f278806e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.370 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f278d971700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27880af200 0x7f27880ab3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.370 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f278d971700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2788156230 0x7f278814a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f278d971700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27881695d0 con 0x7f2788156230 2023-06-23T01:31:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f278d971700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27881571f0 con 0x7f27880af200 2023-06-23T01:31:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f278d971700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2788156600 con 0x7f278806a490 2023-06-23T01:31:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f27877fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2788156230 0x7f278814a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f27877fe700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2788156230 0x7f278814a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57904/0 (socket says 172.21.15.92:57904) 2023-06-23T01:31:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.365+0000 7f27877fe700 1 -- 172.21.15.92:0/3417418799 learned_addr learned my addr 172.21.15.92:0/3417418799 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f277e7fc700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27880af200 0x7f27880ab3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.371 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f2786ffd700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f278806a490 0x7f278806e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f27877fe700 1 -- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f278806a490 msgr2=0x7f278806e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f27877fe700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f278806a490 0x7f278806e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f27877fe700 1 -- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27880af200 msgr2=0x7f27880ab3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f27877fe700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27880af200 0x7f27880ab3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f27877fe700 1 -- 172.21.15.92:0/3417418799 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2780009a20 con 0x7f2788156230 2023-06-23T01:31:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f2786ffd700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f278806a490 0x7f278806e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-06-23T01:31:45.372 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f27877fe700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2788156230 0x7f278814a840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2780012580 tx=0x7f2780019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f277f7fe700 1 -- 172.21.15.92:0/3417418799 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2780027470 con 0x7f2788156230 2023-06-23T01:31:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f277f7fe700 1 -- 172.21.15.92:0/3417418799 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2780027610 con 0x7f2788156230 2023-06-23T01:31:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27881568c0 con 0x7f2788156230 2023-06-23T01:31:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f277f7fe700 1 -- 172.21.15.92:0/3417418799 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27800279d0 con 0x7f2788156230 2023-06-23T01:31:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.366+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2788156e80 con 0x7f2788156230 2023-06-23T01:31:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.367+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2788158800 con 0x7f2788156230 2023-06-23T01:31:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.367+0000 7f277f7fe700 1 -- 172.21.15.92:0/3417418799 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f2780024070 con 0x7f2788156230 2023-06-23T01:31:45.373 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.368+0000 7f277f7fe700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2768066e60 0x7f2768069310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.368+0000 7f277f7fe700 1 -- 172.21.15.92:0/3417418799 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f2780086d50 con 0x7f2788156230 2023-06-23T01:31:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.368+0000 7f2786ffd700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2768066e60 0x7f2768069310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.368+0000 7f2786ffd700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2768066e60 0x7f2768069310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2778001890 tx=0x7f2778009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:45.374 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.372+0000 7f277f7fe700 1 -- 172.21.15.92:0/3417418799 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f278008a030 con 0x7f2788156230 2023-06-23T01:31:45.558 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.557+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f278806cf00 con 0x7f2788156230 2023-06-23T01:31:45.559 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.558+0000 7f277f7fe700 1 -- 172.21.15.92:0/3417418799 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16879 (secure 0 0 0) 0x7f2780053fd0 con 0x7f2788156230 2023-06-23T01:31:45.559 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:31:45.560 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":22,"fsid":"c94811cf-f7e5-4c57-b505-a983532f8da8","created":"2023-06-23T01:31:20.907484+0000","modified":"2023-06-23T01:31:45.208697+0000","last_up_change":"2023-06-23T01:31:32.355093+0000","last_in_change":"2023-06-23T01:31:23.354162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:31:32.414191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:31:39.216467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:31:43.719481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T01:31:44.173018+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"4bcf4cdc-7c98-4a4a-bc56-133f22f4d802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":87954},{"type":"v1","addr":"172.21.15.92:6809","nonce":87954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":87954},{"type":"v1","addr":"172.21.15.92:6811","nonce":87954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":87954},{"type":"v1","addr":"172.21.15.92:6815","nonce":87954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":87954},{"type":"v1","addr":"172.21.15.92:6813","nonce":87954}]},"public_addr":"172.21.15.92:6809/87954","cluster_addr":"172.21.15.92:6811/87954","heartbeat_back_addr":"172.21.15.92:6815/87954","heartbeat_front_addr":"172.21.15.92:6813/87954","state":["exists","up"]},{"osd":1,"uuid":"a2e05a19-0c70-4cb4-aacd-3381219435b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":87955},{"type":"v1","addr":"172.21.15.92:6821","nonce":87955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":87955},{"type":"v1","addr":"172.21.15.92:6826","nonce":87955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":87955},{"type":"v1","addr":"172.21.15.92:6831","nonce":87955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":87955},{"type":"v1","addr":"172.21.15.92:6829","nonce":87955}]},"public_addr":"172.21.15.92:6821/87955","cluster_addr":"172.21.15.92:6826/87955","heartbeat_back_addr":"172.21.15.92:6831/87955","heartbeat_front_addr":"172.21.15.92:6829/87955","state":["exists","up"]},{"osd":2,"uuid":"20f85617-68ff-491f-a436-aa1ad1979595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":87951},{"type":"v1","addr":"172.21.15.92:6801","nonce":87951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":87951},{"type":"v1","addr":"172.21.15.92:6803","nonce":87951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":87951},{"type":"v1","addr":"172.21.15.92:6807","nonce":87951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":87951},{"type":"v1","addr":"172.21.15.92:6805","nonce":87951}]},"public_addr":"172.21.15.92:6801/87951","cluster_addr":"172.21.15.92:6803/87951","heartbeat_back_addr":"172.21.15.92:6807/87951","heartbeat_front_addr":"172.21.15.92:6805/87951","state":["exists","up"]},{"osd":3,"uuid":"3e5b05f4-6b77-4612-9720-38ece0b1388b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":87956},{"type":"v1","addr":"172.21.15.92:6817","nonce":87956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":87956},{"type":"v1","addr":"172.21.15.92:6820","nonce":87956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":87956},{"type":"v1","addr":"172.21.15.92:6827","nonce":87956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":87956},{"type":"v1","addr":"172.21.15.92:6824","nonce":87956}]},"public_addr":"172.21.15.92:6817/87956","cluster_addr":"172.21.15.92:6820/87956","heartbeat_back_addr":"172.21.15.92:6827/87956","heartbeat_front_addr":"172.21.15.92:6824/87956","state":["exists","up"]},{"osd":4,"uuid":"4d94d9c9-4288-477f-801f-c66683295335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6801","nonce":88172},{"type":"v1","addr":"172.21.15.119:6802","nonce":88172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":88172},{"type":"v1","addr":"172.21.15.119:6806","nonce":88172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":88172},{"type":"v1","addr":"172.21.15.119:6814","nonce":88172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":88172},{"type":"v1","addr":"172.21.15.119:6810","nonce":88172}]},"public_addr":"172.21.15.119:6802/88172","cluster_addr":"172.21.15.119:6806/88172","heartbeat_back_addr":"172.21.15.119:6814/88172","heartbeat_front_addr":"172.21.15.119:6810/88172","state":["exists","up"]},{"osd":5,"uuid":"dc6b2f59-8af4-42b2-b7f9-bf36714c358f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":88173},{"type":"v1","addr":"172.21.15.119:6803","nonce":88173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6805","nonce":88173},{"type":"v1","addr":"172.21.15.119:6807","nonce":88173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":88173},{"type":"v1","addr":"172.21.15.119:6815","nonce":88173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":88173},{"type":"v1","addr":"172.21.15.119:6811","nonce":88173}]},"public_addr":"172.21.15.119:6803/88173","cluster_addr":"172.21.15.119:6807/88173","heartbeat_back_addr":"172.21.15.119:6815/88173","heartbeat_front_addr":"172.21.15.119:6811/88173","state":["exists","up"]},{"osd":6,"uuid":"2ac0eb61-39bf-4666-8c24-279bf8b9c51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6817","nonce":88202},{"type":"v1","addr":"172.21.15.119:6819","nonce":88202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6821","nonce":88202},{"type":"v1","addr":"172.21.15.119:6823","nonce":88202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6829","nonce":88202},{"type":"v1","addr":"172.21.15.119:6831","nonce":88202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6825","nonce":88202},{"type":"v1","addr":"172.21.15.119:6827","nonce":88202}]},"public_addr":"172.21.15.119:6819/88202","cluster_addr":"172.21.15.119:6823/88202","heartbeat_back_addr":"172.21.15.119:6831/88202","heartbeat_front_addr":"172.21.15.119:6827/88202","state":["exists","up"]},{"osd":7,"uuid":"c4ff77b0-23dc-477d-adaa-d7fd9108f5da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":88203},{"type":"v1","addr":"172.21.15.119:6818","nonce":88203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":88203},{"type":"v1","addr":"172.21.15.119:6822","nonce":88203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":88203},{"type":"v1","addr":"172.21.15.119:6830","nonce":88203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":88203},{"type":"v1","addr":"172.21.15.119:6826","nonce":88203}]},"public_addr":"172.21.15.119:6818/88203","cluster_addr":"172.21.15.119:6822/88203","heartbeat_back_addr":"172.21.15.119:6830/88203","heartbeat_front_addr":"172.21.15.119:6826/88203","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.542554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.495262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.517893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.443537+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.444113+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.472890+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.556249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.573872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:31:45.561 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2768066e60 msgr2=0x7f2768069310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.561 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2768066e60 0x7f2768069310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2778001890 tx=0x7f2778009040 comp rx=0 tx=0).stop 2023-06-23T01:31:45.561 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2788156230 msgr2=0x7f278814a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2788156230 0x7f278814a840 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2780012580 tx=0x7f2780019000 comp rx=0 tx=0).stop 2023-06-23T01:31:45.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 shutdown_connections 2023-06-23T01:31:45.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f2768066e60 0x7f2768069310 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f2788156230 0x7f278814a840 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f278806a490 0x7f278806e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 --2- 172.21.15.92:0/3417418799 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f27880af200 0x7f27880ab3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.562 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 >> 172.21.15.92:0/3417418799 conn(0x7f2788068fa0 msgr2=0x7f2788152bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:45.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 shutdown_connections 2023-06-23T01:31:45.563 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.560+0000 7f278d971700 1 -- 172.21.15.92:0/3417418799 wait complete. 2023-06-23T01:31:45.573 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-06-23T01:31:45.688 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.687+0000 7fa0b9295700 1 Processor -- start 2023-06-23T01:31:45.688 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.687+0000 7fa0b9295700 1 -- start start 2023-06-23T01:31:45.689 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b9295700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b40ca7a0 0x7fa0b4067410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.690 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b9295700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7fa0b4067950 0x7fa0b4163210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.690 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b9295700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b4067db0 con 0x7fa0b4159640 2023-06-23T01:31:45.690 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b9295700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b406f4a0 con 0x7fa0b4067950 2023-06-23T01:31:45.690 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b9295700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b40cac10 con 0x7fa0b40ca7a0 2023-06-23T01:31:45.690 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b2ffd700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7fa0b4159640 0x7fa0b406e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36136/0 (socket says 172.21.15.92:36136) 2023-06-23T01:31:45.690 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b2ffd700 1 -- 172.21.15.92:0/1879142741 learned_addr learned my addr 172.21.15.92:0/1879142741 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:45.691 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b37fe700 1 --2- 172.21.15.92:0/1879142741 >> v2:172.21.15.119:3300/0 conn(0x7fa0b4067950 0x7fa0b4163210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.691 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b27fc700 1 --2- 172.21.15.92:0/1879142741 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b40ca7a0 0x7fa0b4067410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.691 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b37fe700 1 -- 172.21.15.92:0/1879142741 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b40ca7a0 msgr2=0x7fa0b4067410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:45.691 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b37fe700 1 --2- 172.21.15.92:0/1879142741 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b40ca7a0 0x7fa0b4067410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.691 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b37fe700 1 -- 172.21.15.92:0/1879142741 >> v1:172.21.15.92:6789/0 conn(0x7fa0b4159640 legacy=0x7fa0b406e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.691 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b37fe700 1 -- 172.21.15.92:0/1879142741 --> v2:172.21.15.119:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b40c41a0 con 0x7fa0b4067950 2023-06-23T01:31:45.691 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.688+0000 7fa0b37fe700 1 --2- 172.21.15.92:0/1879142741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b4067950 0x7fa0b4163210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009e60 tx=0x7fa0ac009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6c7ec70a3cbb94a server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:45.692 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.689+0000 7fa0b1ffb700 1 -- 172.21.15.92:0/1879142741 <== mon.1 v2:172.21.15.119:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac00e070 con 0x7fa0b4067950 2023-06-23T01:31:45.692 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.689+0000 7fa0b1ffb700 1 -- 172.21.15.92:0/1879142741 <== mon.1 v2:172.21.15.119:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0ac00bb60 con 0x7fa0b4067950 2023-06-23T01:31:45.693 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.689+0000 7fa0b1ffb700 1 -- 172.21.15.92:0/1879142741 <== mon.1 v2:172.21.15.119:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac012430 con 0x7fa0b4067950 2023-06-23T01:31:45.693 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.690+0000 7fa0b9295700 1 -- 172.21.15.92:0/1879142741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b4067950 msgr2=0x7fa0b4163210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.693 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.690+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1879142741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b4067950 0x7fa0b4163210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009e60 tx=0x7fa0ac009780 comp rx=0 tx=0).stop 2023-06-23T01:31:45.693 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.690+0000 7fa0b9295700 1 -- 172.21.15.92:0/1879142741 shutdown_connections 2023-06-23T01:31:45.693 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.690+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1879142741 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b4067950 0x7fa0b4163210 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.693 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.690+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1879142741 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b40ca7a0 0x7fa0b4067410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.693 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.690+0000 7fa0b9295700 1 -- 172.21.15.92:0/1879142741 >> 172.21.15.92:0/1879142741 conn(0x7fa0b4068520 msgr2=0x7fa0b4068920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:45.694 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.690+0000 7fa0b9295700 1 -- 172.21.15.92:0/1879142741 shutdown_connections 2023-06-23T01:31:45.694 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.690+0000 7fa0b9295700 1 -- 172.21.15.92:0/1879142741 wait complete. 2023-06-23T01:31:45.694 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.691+0000 7fa0b9295700 1 Processor -- start 2023-06-23T01:31:45.694 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.691+0000 7fa0b9295700 1 -- start start 2023-06-23T01:31:45.696 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b9295700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b4067950 0x7fa0b40aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.696 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b9295700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b40ca7a0 0x7fa0b4152d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.696 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b9295700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b4159640 0x7fa0b40c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.696 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b9295700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b4159d30 con 0x7fa0b4067950 2023-06-23T01:31:45.697 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b9295700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b40af070 con 0x7fa0b40ca7a0 2023-06-23T01:31:45.697 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b9295700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b415f960 con 0x7fa0b4159640 2023-06-23T01:31:45.697 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b4067950 0x7fa0b40aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.697 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b4067950 0x7fa0b40aeb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57910/0 (socket says 172.21.15.92:57910) 2023-06-23T01:31:45.697 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 -- 172.21.15.92:0/1996479136 learned_addr learned my addr 172.21.15.92:0/1996479136 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:45.697 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b37fe700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b4159640 0x7fa0b40c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.697 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b27fc700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b40ca7a0 0x7fa0b4152d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.698 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 -- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b4159640 msgr2=0x7fa0b40c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.698 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b4159640 0x7fa0b40c8c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.698 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 -- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b40ca7a0 msgr2=0x7fa0b4152d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.698 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b40ca7a0 0x7fa0b4152d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.698 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 -- 172.21.15.92:0/1996479136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac009a20 con 0x7fa0b4067950 2023-06-23T01:31:45.698 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b27fc700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b40ca7a0 0x7fa0b4152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:31:45.698 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b37fe700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b4159640 0x7fa0b40c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-06-23T01:31:45.698 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.692+0000 7fa0b2ffd700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b4067950 0x7fa0b40aeb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400c590 tx=0x7fa0a4007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:45.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.693+0000 7fa09affd700 1 -- 172.21.15.92:0/1996479136 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4005b30 con 0x7fa0b4067950 2023-06-23T01:31:45.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.693+0000 7fa09affd700 1 -- 172.21.15.92:0/1996479136 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a4005cb0 con 0x7fa0b4067950 2023-06-23T01:31:45.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.693+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b415fc40 con 0x7fa0b4067950 2023-06-23T01:31:45.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.693+0000 7fa09affd700 1 -- 172.21.15.92:0/1996479136 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a4007320 con 0x7fa0b4067950 2023-06-23T01:31:45.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.693+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b4160200 con 0x7fa0b4067950 2023-06-23T01:31:45.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.693+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b404c1b0 con 0x7fa0b4067950 2023-06-23T01:31:45.699 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.694+0000 7fa09affd700 1 -- 172.21.15.92:0/1996479136 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7fa0a401b3b0 con 0x7fa0b4067950 2023-06-23T01:31:45.700 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.694+0000 7fa09affd700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fa0a0066ec0 0x7fa0a0069370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:45.700 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.694+0000 7fa0b27fc700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fa0a0066ec0 0x7fa0a0069370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:45.700 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.694+0000 7fa09affd700 1 -- 172.21.15.92:0/1996479136 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7fa0a4083830 con 0x7fa0b4067950 2023-06-23T01:31:45.700 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.695+0000 7fa0b27fc700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fa0a0066ec0 0x7fa0a0069370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8006f00 tx=0x7fa0a8008040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:45.700 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.698+0000 7fa09affd700 1 -- 172.21.15.92:0/1996479136 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7fa0a4050a00 con 0x7fa0b4067950 2023-06-23T01:31:45.884 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.883+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0b4067db0 con 0x7fa0b4067950 2023-06-23T01:31:45.885 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.884+0000 7fa09affd700 1 -- 172.21.15.92:0/1996479136 <== mon.0 v2:172.21.15.92:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16879 (secure 0 0 0) 0x7fa0b4067db0 con 0x7fa0b4067950 2023-06-23T01:31:45.886 INFO:teuthology.orchestra.run.smithi092.stdout: 2023-06-23T01:31:45.886 INFO:teuthology.orchestra.run.smithi092.stdout:{"epoch":22,"fsid":"c94811cf-f7e5-4c57-b505-a983532f8da8","created":"2023-06-23T01:31:20.907484+0000","modified":"2023-06-23T01:31:45.208697+0000","last_up_change":"2023-06-23T01:31:32.355093+0000","last_in_change":"2023-06-23T01:31:23.354162+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-06-23T01:31:32.414191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-06-23T01:31:39.216467+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2023-06-23T01:31:43.719481+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2023-06-23T01:31:44.173018+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"4bcf4cdc-7c98-4a4a-bc56-133f22f4d802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6808","nonce":87954},{"type":"v1","addr":"172.21.15.92:6809","nonce":87954}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6810","nonce":87954},{"type":"v1","addr":"172.21.15.92:6811","nonce":87954}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6814","nonce":87954},{"type":"v1","addr":"172.21.15.92:6815","nonce":87954}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6812","nonce":87954},{"type":"v1","addr":"172.21.15.92:6813","nonce":87954}]},"public_addr":"172.21.15.92:6809/87954","cluster_addr":"172.21.15.92:6811/87954","heartbeat_back_addr":"172.21.15.92:6815/87954","heartbeat_front_addr":"172.21.15.92:6813/87954","state":["exists","up"]},{"osd":1,"uuid":"a2e05a19-0c70-4cb4-aacd-3381219435b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6819","nonce":87955},{"type":"v1","addr":"172.21.15.92:6821","nonce":87955}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6823","nonce":87955},{"type":"v1","addr":"172.21.15.92:6826","nonce":87955}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6830","nonce":87955},{"type":"v1","addr":"172.21.15.92:6831","nonce":87955}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6828","nonce":87955},{"type":"v1","addr":"172.21.15.92:6829","nonce":87955}]},"public_addr":"172.21.15.92:6821/87955","cluster_addr":"172.21.15.92:6826/87955","heartbeat_back_addr":"172.21.15.92:6831/87955","heartbeat_front_addr":"172.21.15.92:6829/87955","state":["exists","up"]},{"osd":2,"uuid":"20f85617-68ff-491f-a436-aa1ad1979595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6800","nonce":87951},{"type":"v1","addr":"172.21.15.92:6801","nonce":87951}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6802","nonce":87951},{"type":"v1","addr":"172.21.15.92:6803","nonce":87951}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6806","nonce":87951},{"type":"v1","addr":"172.21.15.92:6807","nonce":87951}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6804","nonce":87951},{"type":"v1","addr":"172.21.15.92:6805","nonce":87951}]},"public_addr":"172.21.15.92:6801/87951","cluster_addr":"172.21.15.92:6803/87951","heartbeat_back_addr":"172.21.15.92:6807/87951","heartbeat_front_addr":"172.21.15.92:6805/87951","state":["exists","up"]},{"osd":3,"uuid":"3e5b05f4-6b77-4612-9720-38ece0b1388b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6816","nonce":87956},{"type":"v1","addr":"172.21.15.92:6817","nonce":87956}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6818","nonce":87956},{"type":"v1","addr":"172.21.15.92:6820","nonce":87956}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6825","nonce":87956},{"type":"v1","addr":"172.21.15.92:6827","nonce":87956}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.92:6822","nonce":87956},{"type":"v1","addr":"172.21.15.92:6824","nonce":87956}]},"public_addr":"172.21.15.92:6817/87956","cluster_addr":"172.21.15.92:6820/87956","heartbeat_back_addr":"172.21.15.92:6827/87956","heartbeat_front_addr":"172.21.15.92:6824/87956","state":["exists","up"]},{"osd":4,"uuid":"4d94d9c9-4288-477f-801f-c66683295335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6801","nonce":88172},{"type":"v1","addr":"172.21.15.119:6802","nonce":88172}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6804","nonce":88172},{"type":"v1","addr":"172.21.15.119:6806","nonce":88172}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6812","nonce":88172},{"type":"v1","addr":"172.21.15.119:6814","nonce":88172}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6808","nonce":88172},{"type":"v1","addr":"172.21.15.119:6810","nonce":88172}]},"public_addr":"172.21.15.119:6802/88172","cluster_addr":"172.21.15.119:6806/88172","heartbeat_back_addr":"172.21.15.119:6814/88172","heartbeat_front_addr":"172.21.15.119:6810/88172","state":["exists","up"]},{"osd":5,"uuid":"dc6b2f59-8af4-42b2-b7f9-bf36714c358f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6800","nonce":88173},{"type":"v1","addr":"172.21.15.119:6803","nonce":88173}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6805","nonce":88173},{"type":"v1","addr":"172.21.15.119:6807","nonce":88173}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6813","nonce":88173},{"type":"v1","addr":"172.21.15.119:6815","nonce":88173}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6809","nonce":88173},{"type":"v1","addr":"172.21.15.119:6811","nonce":88173}]},"public_addr":"172.21.15.119:6803/88173","cluster_addr":"172.21.15.119:6807/88173","heartbeat_back_addr":"172.21.15.119:6815/88173","heartbeat_front_addr":"172.21.15.119:6811/88173","state":["exists","up"]},{"osd":6,"uuid":"2ac0eb61-39bf-4666-8c24-279bf8b9c51d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6817","nonce":88202},{"type":"v1","addr":"172.21.15.119:6819","nonce":88202}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6821","nonce":88202},{"type":"v1","addr":"172.21.15.119:6823","nonce":88202}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6829","nonce":88202},{"type":"v1","addr":"172.21.15.119:6831","nonce":88202}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6825","nonce":88202},{"type":"v1","addr":"172.21.15.119:6827","nonce":88202}]},"public_addr":"172.21.15.119:6819/88202","cluster_addr":"172.21.15.119:6823/88202","heartbeat_back_addr":"172.21.15.119:6831/88202","heartbeat_front_addr":"172.21.15.119:6827/88202","state":["exists","up"]},{"osd":7,"uuid":"c4ff77b0-23dc-477d-adaa-d7fd9108f5da","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6816","nonce":88203},{"type":"v1","addr":"172.21.15.119:6818","nonce":88203}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6820","nonce":88203},{"type":"v1","addr":"172.21.15.119:6822","nonce":88203}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6828","nonce":88203},{"type":"v1","addr":"172.21.15.119:6830","nonce":88203}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.119:6824","nonce":88203},{"type":"v1","addr":"172.21.15.119:6826","nonce":88203}]},"public_addr":"172.21.15.119:6818/88203","cluster_addr":"172.21.15.119:6822/88203","heartbeat_back_addr":"172.21.15.119:6830/88203","heartbeat_front_addr":"172.21.15.119:6826/88203","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.542554+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.495262+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.517893+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.443537+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.444113+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.472890+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.556249+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-06-23T01:31:28.573872+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-06-23T01:31:45.888 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fa0a0066ec0 msgr2=0x7fa0a0069370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.888 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fa0a0066ec0 0x7fa0a0069370 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8006f00 tx=0x7fa0a8008040 comp rx=0 tx=0).stop 2023-06-23T01:31:45.888 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b4067950 msgr2=0x7fa0b40aeb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:45.888 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b4067950 0x7fa0b40aeb30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400c590 tx=0x7fa0a4007f90 comp rx=0 tx=0).stop 2023-06-23T01:31:45.889 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 shutdown_connections 2023-06-23T01:31:45.889 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7fa0a0066ec0 0x7fa0a0069370 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.889 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7fa0b4159640 0x7fa0b40c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.889 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7fa0b4067950 0x7fa0b40aeb30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.889 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 --2- 172.21.15.92:0/1996479136 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7fa0b40ca7a0 0x7fa0b4152d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:45.889 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.886+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 >> 172.21.15.92:0/1996479136 conn(0x7fa0b4068520 msgr2=0x7fa0b40b0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:45.889 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.887+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 shutdown_connections 2023-06-23T01:31:45.889 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:45.887+0000 7fa0b9295700 1 -- 172.21.15.92:0/1996479136 wait complete. 2023-06-23T01:31:45.900 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-06-23T01:31:46.016 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.015+0000 7efdabef1700 1 Processor -- start 2023-06-23T01:31:46.016 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.015+0000 7efdabef1700 1 -- start start 2023-06-23T01:31:46.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efdabef1700 1 --2- >> v2:172.21.15.119:3300/0 conn(0x7efda4153e50 0x7efda40d0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efdabef1700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda40cedd0 con 0x7efda4153a80 2023-06-23T01:31:46.025 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efdabef1700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efda415aed0 con 0x7efda40b2010 2023-06-23T01:31:46.026 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efdabef1700 1 -- --> v2:172.21.15.119:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efda40d14c0 con 0x7efda4153e50 2023-06-23T01:31:46.026 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda9c8d700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7efda4153a80 0x7efda415a7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36138/0 (socket says 172.21.15.92:36138) 2023-06-23T01:31:46.026 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda9c8d700 1 -- 172.21.15.92:0/1094135142 learned_addr learned my addr 172.21.15.92:0/1094135142 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:46.027 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534769342 0 0) 0x7efda40cedd0 con 0x7efda4153a80 2023-06-23T01:31:46.027 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 --> v1:172.21.15.92:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd8c003570 con 0x7efda4153a80 2023-06-23T01:31:46.027 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda948c700 1 --2- 172.21.15.92:0/1094135142 >> v2:172.21.15.119:3300/0 conn(0x7efda4153e50 0x7efda40d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.028 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 <== mon.0 v1:172.21.15.92:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 484677142 0 0) 0x7efd8c003570 con 0x7efda4153a80 2023-06-23T01:31:46.028 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 >> v2:172.21.15.119:3300/0 conn(0x7efda4153e50 msgr2=0x7efda40d0590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:46.028 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 --2- 172.21.15.92:0/1094135142 >> v2:172.21.15.119:3300/0 conn(0x7efda4153e50 0x7efda40d0590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.028 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 >> v1:172.21.15.119:6789/0 conn(0x7efda40b2010 legacy=0x7efda40cf260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.028 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 --> v1:172.21.15.92:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda40ceac0 con 0x7efda4153a80 2023-06-23T01:31:46.028 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 <== mon.0 v1:172.21.15.92:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3236472324 0 0) 0x7efd9c001470 con 0x7efda4153a80 2023-06-23T01:31:46.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 >> v1:172.21.15.92:6789/0 conn(0x7efda4153a80 legacy=0x7efda415a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c003a60 0x7efd8c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efd8c004810 0x7efd8c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd8c0051c0 0x7efd8c00ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda40d14c0 con 0x7efd8c0051c0 2023-06-23T01:31:46.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c00c160 con 0x7efd8c003a60 2023-06-23T01:31:46.029 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.016+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd8c00c320 con 0x7efd8c004810 2023-06-23T01:31:46.030 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efdaa48e700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd8c0051c0 0x7efd8c00ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.030 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efda948c700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efd8c004810 0x7efd8c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.030 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efda9c8d700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c003a60 0x7efd8c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.030 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efdaa48e700 1 -- 172.21.15.92:0/1094135142 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efd8c004810 msgr2=0x7efd8c004c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:46.030 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efdaa48e700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efd8c004810 0x7efd8c004c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.030 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efdaa48e700 1 -- 172.21.15.92:0/1094135142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c003a60 msgr2=0x7efd8c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:46.031 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efdaa48e700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c003a60 0x7efd8c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.031 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efdaa48e700 1 -- 172.21.15.92:0/1094135142 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd8c003570 con 0x7efd8c0051c0 2023-06-23T01:31:46.031 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efdaa48e700 1 -- 172.21.15.92:0/1094135142 reap_dead start 2023-06-23T01:31:46.031 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efdaa48e700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd8c0051c0 0x7efd8c00ba60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efd94001fc0 tx=0x7efd94001640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e3e000157ef2262 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:46.031 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94001950 con 0x7efd8c0051c0 2023-06-23T01:31:46.031 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9400c8b0 con 0x7efd8c0051c0 2023-06-23T01:31:46.032 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.017+0000 7efda8c8b700 1 -- 172.21.15.92:0/1094135142 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9400cc90 con 0x7efd8c0051c0 2023-06-23T01:31:46.032 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.018+0000 7efdabef1700 1 -- 172.21.15.92:0/1094135142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd8c0051c0 msgr2=0x7efd8c00ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.032 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.018+0000 7efdabef1700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd8c0051c0 0x7efd8c00ba60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7efd94001fc0 tx=0x7efd94001640 comp rx=0 tx=0).stop 2023-06-23T01:31:46.032 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.018+0000 7efdabef1700 1 -- 172.21.15.92:0/1094135142 shutdown_connections 2023-06-23T01:31:46.032 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.018+0000 7efdabef1700 1 --2- 172.21.15.92:0/1094135142 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efd8c0051c0 0x7efd8c00ba60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.032 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.018+0000 7efdabef1700 1 -- 172.21.15.92:0/1094135142 >> 172.21.15.92:0/1094135142 conn(0x7efda4070290 msgr2=0x7efda4070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:46.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.018+0000 7efdabef1700 1 -- 172.21.15.92:0/1094135142 shutdown_connections 2023-06-23T01:31:46.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.018+0000 7efdabef1700 1 -- 172.21.15.92:0/1094135142 wait complete. 2023-06-23T01:31:46.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.019+0000 7efdabef1700 1 Processor -- start 2023-06-23T01:31:46.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.019+0000 7efdabef1700 1 -- start start 2023-06-23T01:31:46.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdabef1700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0051c0 0x7efda4155cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdabef1700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efda414ffa0 0x7efda4157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.033 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdabef1700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda4157880 0x7efda4169d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdabef1700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda4156230 con 0x7efda4157880 2023-06-23T01:31:46.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdabef1700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efda4157cf0 con 0x7efd8c0051c0 2023-06-23T01:31:46.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdabef1700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efda4155860 con 0x7efda414ffa0 2023-06-23T01:31:46.034 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda4157880 0x7efda4169d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.035 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda4157880 0x7efda4169d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57922/0 (socket says 172.21.15.92:57922) 2023-06-23T01:31:46.035 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 -- 172.21.15.92:0/70732521 learned_addr learned my addr 172.21.15.92:0/70732521 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:46.035 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efda9c8d700 1 --2- 172.21.15.92:0/70732521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0051c0 0x7efda4155cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.035 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efda948c700 1 --2- 172.21.15.92:0/70732521 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efda414ffa0 0x7efda4157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.035 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 -- 172.21.15.92:0/70732521 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efda414ffa0 msgr2=0x7efda4157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.036 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 --2- 172.21.15.92:0/70732521 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7efda414ffa0 0x7efda4157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.036 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 -- 172.21.15.92:0/70732521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0051c0 msgr2=0x7efda4155cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.036 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 --2- 172.21.15.92:0/70732521 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7efd8c0051c0 0x7efda4155cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.037 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 -- 172.21.15.92:0/70732521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efda4154c50 con 0x7efda4157880 2023-06-23T01:31:46.037 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdaa48e700 1 --2- 172.21.15.92:0/70732521 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7efda4157880 0x7efda4169d30 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7efd9400c6a0 tx=0x7efd94015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:46.037 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efda1ffb700 1 -- 172.21.15.92:0/70732521 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd94001950 con 0x7efda4157880 2023-06-23T01:31:46.037 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efda1ffb700 1 -- 172.21.15.92:0/70732521 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd9400c6d0 con 0x7efda4157880 2023-06-23T01:31:46.037 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efda1ffb700 1 -- 172.21.15.92:0/70732521 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efd9401e850 con 0x7efda4157880 2023-06-23T01:31:46.038 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdabef1700 1 -- 172.21.15.92:0/70732521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efda416a950 con 0x7efda4157880 2023-06-23T01:31:46.038 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.020+0000 7efdabef1700 1 -- 172.21.15.92:0/70732521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efda4155210 con 0x7efda4157880 2023-06-23T01:31:46.038 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.021+0000 7efdabef1700 1 -- 172.21.15.92:0/70732521 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efda404c1b0 con 0x7efda4157880 2023-06-23T01:31:46.038 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.022+0000 7efda1ffb700 1 -- 172.21.15.92:0/70732521 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7efd9401ec40 con 0x7efda4157880 2023-06-23T01:31:46.038 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.022+0000 7efda1ffb700 1 --2- 172.21.15.92:0/70732521 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7efd90066e90 0x7efd90069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.038 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.022+0000 7efda1ffb700 1 -- 172.21.15.92:0/70732521 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6058+0+0 (secure 0 0 0) 0x7efd940859b0 con 0x7efda4157880 2023-06-23T01:31:46.039 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.022+0000 7efda9c8d700 1 --2- 172.21.15.92:0/70732521 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7efd90066e90 0x7efd90069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.039 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.023+0000 7efda9c8d700 1 --2- 172.21.15.92:0/70732521 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7efd90066e90 0x7efd90069340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7efd9c000c20 tx=0x7efd9c001e50 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:46.039 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.026+0000 7efda1ffb700 1 -- 172.21.15.92:0/70732521 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7efd94052c40 con 0x7efda4157880 2023-06-23T01:31:46.257 INFO:teuthology.orchestra.run.smithi092.stderr:no valid command found; 1 closest matches: 2023-06-23T01:31:46.257 INFO:teuthology.orchestra.run.smithi092.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:31:46.257 INFO:teuthology.orchestra.run.smithi092.stderr:Error EINVAL: invalid command 2023-06-23T01:31:46.261 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:31:46.262 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-06-23T01:31:46.262 DEBUG:teuthology.orchestra.run.smithi092:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-06-23T01:31:46.383 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.377+0000 7f46fb59e700 1 Processor -- start 2023-06-23T01:31:46.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.377+0000 7f46fb59e700 1 -- start start 2023-06-23T01:31:46.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fb59e700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f46fc0b07f0 0x7f46fc1596e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fb59e700 1 -- --> v1:172.21.15.92:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc0cb7d0 con 0x7f46fc0b23d0 2023-06-23T01:31:46.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fb59e700 1 -- --> v1:172.21.15.119:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc0c72b0 con 0x7f46fc159310 2023-06-23T01:31:46.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fb59e700 1 -- --> v2:172.21.15.92:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc0b2e10 con 0x7f46fc0b07f0 2023-06-23T01:31:46.384 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fad9d700 1 --2- >> v2:172.21.15.92:3300/0 conn(0x7f46fc0b07f0 0x7f46fc1596e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.385 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46f1d9b700 1 --1- >> v1:172.21.15.92:6789/0 conn(0x7f46fc0b23d0 0x7f46fc0b00e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.92:6789/0 says I am v1:172.21.15.92:36140/0 (socket says 172.21.15.92:36140) 2023-06-23T01:31:46.385 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46f1d9b700 1 -- 172.21.15.92:0/1293455186 learned_addr learned my addr 172.21.15.92:0/1293455186 (peer_addr_for_me v1:172.21.15.92:0/0) 2023-06-23T01:31:46.385 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fad9d700 1 -- 172.21.15.92:0/1293455186 >> v1:172.21.15.119:6789/0 conn(0x7f46fc159310 legacy=0x7f46fc15dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-06-23T01:31:46.385 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fad9d700 1 -- 172.21.15.92:0/1293455186 >> v1:172.21.15.92:6789/0 conn(0x7f46fc0b23d0 legacy=0x7f46fc0b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.385 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46f9d9b700 1 -- 172.21.15.92:0/1293455186 <== mon.0 v1:172.21.15.92:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202484107 0 0) 0x7f46fc0cb7d0 con 0x7f46fc0b23d0 2023-06-23T01:31:46.385 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fad9d700 1 -- 172.21.15.92:0/1293455186 --> v2:172.21.15.92:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0b2850 con 0x7f46fc0b07f0 2023-06-23T01:31:46.385 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.378+0000 7f46fad9d700 1 --2- 172.21.15.92:0/1293455186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46fc0b07f0 0x7f46fc1596e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009e60 tx=0x7f46f4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eddd209de9a00dcc server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:46.386 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46f9d9b700 1 -- 172.21.15.92:0/1293455186 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f400e070 con 0x7f46fc0b07f0 2023-06-23T01:31:46.386 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46f9d9b700 1 -- 172.21.15.92:0/1293455186 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f400bbf0 con 0x7f46fc0b07f0 2023-06-23T01:31:46.386 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46f9d9b700 1 -- 172.21.15.92:0/1293455186 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4012430 con 0x7f46fc0b07f0 2023-06-23T01:31:46.386 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46fb59e700 1 -- 172.21.15.92:0/1293455186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46fc0b07f0 msgr2=0x7f46fc1596e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.386 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46fb59e700 1 --2- 172.21.15.92:0/1293455186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46fc0b07f0 0x7f46fc1596e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f46f4009e60 tx=0x7f46f4009780 comp rx=0 tx=0).stop 2023-06-23T01:31:46.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46fb59e700 1 -- 172.21.15.92:0/1293455186 shutdown_connections 2023-06-23T01:31:46.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46fb59e700 1 --2- 172.21.15.92:0/1293455186 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46fc0b07f0 0x7f46fc1596e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46fb59e700 1 -- 172.21.15.92:0/1293455186 >> 172.21.15.92:0/1293455186 conn(0x7f46fc070290 msgr2=0x7f46fc070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-06-23T01:31:46.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46fb59e700 1 -- 172.21.15.92:0/1293455186 shutdown_connections 2023-06-23T01:31:46.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.379+0000 7f46fb59e700 1 -- 172.21.15.92:0/1293455186 wait complete. 2023-06-23T01:31:46.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.380+0000 7f46fb59e700 1 Processor -- start 2023-06-23T01:31:46.387 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.380+0000 7f46fb59e700 1 -- start start 2023-06-23T01:31:46.388 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46fb59e700 1 --2- >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc0b07f0 0x7f46fc155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.388 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46fb59e700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46fc159310 0x7f46fc155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.388 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46fb59e700 1 --2- >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f46fc0b4a30 0x7f46fc0b4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.388 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46fb59e700 1 -- --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0c83a0 con 0x7f46fc159310 2023-06-23T01:31:46.388 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46fb59e700 1 -- --> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc0b53e0 con 0x7f46fc0b07f0 2023-06-23T01:31:46.388 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46fb59e700 1 -- --> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc1694e0 con 0x7f46fc0b4a30 2023-06-23T01:31:46.388 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46fc159310 0x7f46fc155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.388 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 --2- >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46fc159310 0x7f46fc155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.92:3300/0 says I am v2:172.21.15.92:57944/0 (socket says 172.21.15.92:57944) 2023-06-23T01:31:46.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 -- 172.21.15.92:0/109856655 learned_addr learned my addr 172.21.15.92:0/109856655 (peer_addr_for_me v2:172.21.15.92:0/0) 2023-06-23T01:31:46.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46fa59c700 1 --2- 172.21.15.92:0/109856655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc0b07f0 0x7f46fc155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46fad9d700 1 --2- 172.21.15.92:0/109856655 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f46fc0b4a30 0x7f46fc0b4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 -- 172.21.15.92:0/109856655 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f46fc0b4a30 msgr2=0x7f46fc0b4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 --2- 172.21.15.92:0/109856655 >> [v2:172.21.15.119:3301/0,v1:172.21.15.119:6790/0] conn(0x7f46fc0b4a30 0x7f46fc0b4ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 -- 172.21.15.92:0/109856655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc0b07f0 msgr2=0x7f46fc155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-06-23T01:31:46.389 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 --2- 172.21.15.92:0/109856655 >> [v2:172.21.15.119:3300/0,v1:172.21.15.119:6789/0] conn(0x7f46fc0b07f0 0x7f46fc155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-06-23T01:31:46.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 -- 172.21.15.92:0/109856655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f4009a20 con 0x7f46fc159310 2023-06-23T01:31:46.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.381+0000 7f46f1d9b700 1 --2- 172.21.15.92:0/109856655 >> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] conn(0x7f46fc159310 0x7f46fc155840 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f46e400c7e0 tx=0x7f46e4007df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:46.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.382+0000 7f46f2ffd700 1 -- 172.21.15.92:0/109856655 <== mon.0 v2:172.21.15.92:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4005b30 con 0x7f46fc159310 2023-06-23T01:31:46.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.382+0000 7f46f2ffd700 1 -- 172.21.15.92:0/109856655 <== mon.0 v2:172.21.15.92:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e4005c90 con 0x7f46fc159310 2023-06-23T01:31:46.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.382+0000 7f46fb59e700 1 -- 172.21.15.92:0/109856655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc0c96b0 con 0x7f46fc159310 2023-06-23T01:31:46.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.382+0000 7f46f2ffd700 1 -- 172.21.15.92:0/109856655 <== mon.0 v2:172.21.15.92:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46e4007300 con 0x7f46fc159310 2023-06-23T01:31:46.390 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.382+0000 7f46fb59e700 1 -- 172.21.15.92:0/109856655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc0c80c0 con 0x7f46fc159310 2023-06-23T01:31:46.391 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.383+0000 7f46fb59e700 1 -- 172.21.15.92:0/109856655 --> [v2:172.21.15.92:3300/0,v1:172.21.15.92:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc169650 con 0x7f46fc159310 2023-06-23T01:31:46.391 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.383+0000 7f46f2ffd700 1 -- 172.21.15.92:0/109856655 <== mon.0 v2:172.21.15.92:3300/0 4 ==== mgrmap(e 5) v1 ==== 84648+0+0 (secure 0 0 0) 0x7f46e40076b0 con 0x7f46fc159310 2023-06-23T01:31:46.392 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.383+0000 7f46f2ffd700 1 --2- 172.21.15.92:0/109856655 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f46dc066e90 0x7f46dc069340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-06-23T01:31:46.392 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.383+0000 7f46fa59c700 1 --2- 172.21.15.92:0/109856655 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f46dc066e90 0x7f46dc069340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-06-23T01:31:46.392 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.383+0000 7f46f2ffd700 1 -- 172.21.15.92:0/109856655 <== mon.0 v2:172.21.15.92:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6058+0+0 (secure 0 0 0) 0x7f46e4084040 con 0x7f46fc159310 2023-06-23T01:31:46.392 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.384+0000 7f46fa59c700 1 --2- 172.21.15.92:0/109856655 >> [v2:172.21.15.92:6832/87719,v1:172.21.15.92:6833/87719] conn(0x7f46dc066e90 0x7f46dc069340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f46ec003870 tx=0x7f46ec009040 comp rx=0 tx=0).ready entity=mgr.4108 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-06-23T01:31:46.392 INFO:teuthology.orchestra.run.smithi092.stderr:2023-06-23T01:31:46.387+0000 7f46f2ffd700 1 -- 172.21.15.92:0/109856655 <== mon.0 v2:172.21.15.92:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173437 (secure 0 0 0) 0x7f46e4051290 con 0x7f46fc159310 2023-06-23T01:31:46.616 INFO:teuthology.orchestra.run.smithi092.stderr:no valid command found; 1 closest matches: 2023-06-23T01:31:46.617 INFO:teuthology.orchestra.run.smithi092.stderr:fs set |refuse_client_session|max_xattr_size [--yes-i-really-mean-it] [--yes-i-really-really-mean-it] 2023-06-23T01:31:46.617 INFO:teuthology.orchestra.run.smithi092.stderr:Error EINVAL: invalid command 2023-06-23T01:31:46.619 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-06-23T01:31:46.619 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:31:46.620 INFO:teuthology.misc:Shutting down mds daemons... 2023-06-23T01:31:46.621 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-06-23T01:31:46.621 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:46.629 INFO:tasks.ceph.mds.a:Stopped 2023-06-23T01:31:46.630 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-06-23T01:31:46.631 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:46.678 INFO:tasks.ceph.mds.c:Stopped 2023-06-23T01:31:46.679 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-06-23T01:31:46.679 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:46.689 INFO:tasks.ceph.mds.b:Stopped 2023-06-23T01:31:46.691 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-06-23T01:31:46.691 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:46.740 INFO:tasks.ceph.mds.d:Stopped 2023-06-23T01:31:46.741 INFO:teuthology.misc:Shutting down osd daemons... 2023-06-23T01:31:46.742 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-06-23T01:31:46.742 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:46.779 INFO:tasks.ceph.osd.0:Stopped 2023-06-23T01:31:46.780 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-06-23T01:31:46.781 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:46.845 INFO:tasks.ceph.osd.1:Stopped 2023-06-23T01:31:46.845 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-06-23T01:31:46.846 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:46.909 INFO:tasks.ceph.osd.2:Stopped 2023-06-23T01:31:46.909 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-06-23T01:31:46.910 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:46.972 INFO:tasks.ceph.osd.3:Stopped 2023-06-23T01:31:46.973 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-06-23T01:31:46.973 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.013 INFO:tasks.ceph.osd.4:Stopped 2023-06-23T01:31:47.014 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-06-23T01:31:47.015 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.086 INFO:tasks.ceph.osd.5:Stopped 2023-06-23T01:31:47.087 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-06-23T01:31:47.087 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.155 INFO:tasks.ceph.osd.6:Stopped 2023-06-23T01:31:47.156 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-06-23T01:31:47.157 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.221 INFO:tasks.ceph.osd.7:Stopped 2023-06-23T01:31:47.221 INFO:teuthology.misc:Shutting down mgr daemons... 2023-06-23T01:31:47.222 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-06-23T01:31:47.222 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.246 INFO:tasks.ceph.mgr.y:Stopped 2023-06-23T01:31:47.247 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-06-23T01:31:47.247 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.285 INFO:tasks.ceph.mgr.x:Stopped 2023-06-23T01:31:47.286 INFO:teuthology.misc:Shutting down mon daemons... 2023-06-23T01:31:47.287 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-06-23T01:31:47.288 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.297 INFO:tasks.ceph.mon.a:Stopped 2023-06-23T01:31:47.298 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-06-23T01:31:47.298 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.335 INFO:tasks.ceph.mon.b:Stopped 2023-06-23T01:31:47.336 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-06-23T01:31:47.337 INFO:teuthology.orchestra.run:waiting for 300 2023-06-23T01:31:47.386 INFO:tasks.ceph.mon.c:Stopped 2023-06-23T01:31:47.387 INFO:tasks.ceph:Checking cluster log for badness... 2023-06-23T01:31:47.388 DEBUG:teuthology.orchestra.run.smithi092:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'OSD full dropping all updates' | egrep -v 'OSD near full' | egrep -v 'pausewr flag' | egrep -v 'failsafe engaged, dropping updates' | egrep -v 'failsafe disengaged, no longer dropping' | egrep -v 'is full \(reached quota' | egrep -v POOL_FULL | egrep -v POOL_BACKFILLFULL | head -n 1 2023-06-23T01:31:47.418 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:31:47.419 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-06-23T01:31:47.822 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:31:47.823 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-06-23T01:31:47.918 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:31:47.918 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-06-23T01:31:47.977 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:31:47.978 DEBUG:teuthology.orchestra.run.smithi092:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-06-23T01:31:48.078 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:48.079 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-06-23T01:31:48.390 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:48.390 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-06-23T01:31:48.478 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:48.479 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-06-23T01:31:48.553 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:48.554 DEBUG:teuthology.orchestra.run.smithi119:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-06-23T01:31:48.641 INFO:tasks.ceph:Archiving mon data... 2023-06-23T01:31:48.642 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/data/mon.a.tgz 2023-06-23T01:31:48.643 DEBUG:teuthology.orchestra.run.smithi092:> mktemp 2023-06-23T01:31:48.659 INFO:teuthology.orchestra.run.smithi092.stdout:/tmp/tmp.8ew5fEgHqz 2023-06-23T01:31:48.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.8ew5fEgHqz 2023-06-23T01:31:48.765 DEBUG:teuthology.orchestra.run.smithi092:> sudo chmod 0666 /tmp/tmp.8ew5fEgHqz 2023-06-23T01:31:48.846 DEBUG:teuthology.orchestra.remote:smithi092:/tmp/tmp.8ew5fEgHqz is 199KB 2023-06-23T01:31:48.896 DEBUG:teuthology.orchestra.run.smithi092:> rm -fr /tmp/tmp.8ew5fEgHqz 2023-06-23T01:31:48.910 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/data/mon.b.tgz 2023-06-23T01:31:48.911 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2023-06-23T01:31:48.927 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.uBiabUJg5S 2023-06-23T01:31:48.927 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.uBiabUJg5S 2023-06-23T01:31:49.086 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.uBiabUJg5S 2023-06-23T01:31:49.167 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.uBiabUJg5S is 278KB 2023-06-23T01:31:49.262 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.uBiabUJg5S 2023-06-23T01:31:49.277 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/data/mon.c.tgz 2023-06-23T01:31:49.277 DEBUG:teuthology.orchestra.run.smithi119:> mktemp 2023-06-23T01:31:49.330 INFO:teuthology.orchestra.run.smithi119.stdout:/tmp/tmp.JkOQM3yxcE 2023-06-23T01:31:49.330 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.JkOQM3yxcE 2023-06-23T01:31:49.438 DEBUG:teuthology.orchestra.run.smithi119:> sudo chmod 0666 /tmp/tmp.JkOQM3yxcE 2023-06-23T01:31:49.518 DEBUG:teuthology.orchestra.remote:smithi119:/tmp/tmp.JkOQM3yxcE is 278KB 2023-06-23T01:31:49.581 DEBUG:teuthology.orchestra.run.smithi119:> rm -fr /tmp/tmp.JkOQM3yxcE 2023-06-23T01:31:49.596 INFO:tasks.ceph:Cleaning ceph cluster... 2023-06-23T01:31:49.597 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T01:31:49.600 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-06-23T01:31:49.660 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-06-23T01:31:49.660 DEBUG:teuthology.orchestra.run.smithi092:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T01:31:49.666 DEBUG:teuthology.orchestra.run.smithi119:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-06-23T01:31:49.689 INFO:teuthology.orchestra.run.smithi092.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T01:31:49.725 INFO:teuthology.orchestra.run.smithi119.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-06-23T01:31:49.731 INFO:tasks.ceph:Archiving crash dumps... 2023-06-23T01:31:49.742 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/remote/smithi092/crash 2023-06-23T01:31:49.745 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T01:31:49.770 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/remote/smithi119/crash 2023-06-23T01:31:49.777 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-06-23T01:31:49.802 INFO:tasks.ceph:Shutting down logrotate 2023-06-23T01:31:49.803 DEBUG:teuthology.orchestra.run.smithi092:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:31:49.821 DEBUG:teuthology.orchestra.run.smithi119:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-06-23T01:31:49.896 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T01:31:49.921 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-06-23T01:31:49.945 INFO:tasks.ceph:Compressing logs... 2023-06-23T01:31:49.946 DEBUG:teuthology.orchestra.run.smithi092:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:31:49.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:31:50.229 INFO:tasks.ceph:Archiving logs... 2023-06-23T01:31:50.230 DEBUG:teuthology.misc:Transferring archived files from smithi092:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/remote/smithi092/log 2023-06-23T01:31:50.231 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:31:50.312 DEBUG:teuthology.misc:Transferring archived files from smithi119:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/remote/smithi119/log 2023-06-23T01:31:50.314 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /var/log/ceph -- . 2023-06-23T01:31:50.374 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:31:50.445 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=fa8f36015b58456581e8a1ae62ba5836 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 1893, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph.py", line 450, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 514, in __init__ self.create() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 717, in create self.set_allow_standby_replay(standby_replay) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 589, in set_allow_standby_replay self.set_var("allow_standby_replay", yes) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/cephfs/filesystem.py", line 574, in set_var self.mon_manager.raw_cluster_cmd("fs", "set", self.name, var, *a) File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1624, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_42ddc46e6e6582d8e29a5a982a81886fe028be2c/qa/tasks/ceph_manager.py", line 1615, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/remote.py", line 525, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_076bbebc42a14f7d568aaa78eabb0038327bcb23/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi092 with status 22: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false' 2023-06-23T01:31:50.451 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-06-23T01:31:50.459 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-06-23T01:31:50.530 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-06-23T01:31:50.531 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:31:50.552 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-06-23T01:31:50.633 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-06-23T01:31:50.641 INFO:teuthology.task.clock:Checking final clock skew... 2023-06-23T01:31:50.642 DEBUG:teuthology.orchestra.run.smithi092:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:31:50.644 DEBUG:teuthology.orchestra.run.smithi119:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-06-23T01:31:50.655 INFO:teuthology.orchestra.run.smithi092.stderr:bash: ntpq: command not found 2023-06-23T01:31:50.657 INFO:teuthology.orchestra.run.smithi119.stderr:bash: ntpq: command not found 2023-06-23T01:31:50.659 INFO:teuthology.orchestra.run.smithi092.stdout:210 Number of sources = 4 2023-06-23T01:31:50.659 INFO:teuthology.orchestra.run.smithi092.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:31:50.659 INFO:teuthology.orchestra.run.smithi092.stdout:=============================================================================== 2023-06-23T01:31:50.659 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv01.front.sepia.ceph.com 2 6 177 0 +2401us[+2395us] +/- 106ms 2023-06-23T01:31:50.660 INFO:teuthology.orchestra.run.smithi092.stdout:^* hv02.front.sepia.ceph.com 3 6 177 0 -1582us[-1587us] +/- 60ms 2023-06-23T01:31:50.660 INFO:teuthology.orchestra.run.smithi092.stdout:^+ hv03.front.sepia.ceph.com 3 6 77 64 +343us[ +337us] +/- 76ms 2023-06-23T01:31:50.660 INFO:teuthology.orchestra.run.smithi092.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:31:50.662 INFO:teuthology.orchestra.run.smithi119.stdout:210 Number of sources = 4 2023-06-23T01:31:50.662 INFO:teuthology.orchestra.run.smithi119.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-06-23T01:31:50.662 INFO:teuthology.orchestra.run.smithi119.stdout:=============================================================================== 2023-06-23T01:31:50.662 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv01.front.sepia.ceph.com 2 6 177 0 +2399us[+2399us] +/- 106ms 2023-06-23T01:31:50.663 INFO:teuthology.orchestra.run.smithi119.stdout:^* hv02.front.sepia.ceph.com 3 6 77 64 -1579us[-1603us] +/- 59ms 2023-06-23T01:31:50.663 INFO:teuthology.orchestra.run.smithi119.stdout:^+ hv03.front.sepia.ceph.com 3 6 177 0 +343us[ +343us] +/- 77ms 2023-06-23T01:31:50.663 INFO:teuthology.orchestra.run.smithi119.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-06-23T01:31:50.663 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-06-23T01:31:50.671 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-06-23T01:31:50.672 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-06-23T01:31:50.680 DEBUG:teuthology.orchestra.run.smithi092:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:31:50.786 DEBUG:teuthology.orchestra.run.smithi119:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-06-23T01:31:50.871 DEBUG:teuthology.orchestra.run.smithi092:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:31:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:31:50.901 DEBUG:teuthology.orchestra.run.smithi119:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-06-23T01:31:50.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:31:50.930 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-06-23T01:31:50.944 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-06-23T01:31:50.952 INFO:teuthology.task.internal:Duration was 694.257863 seconds 2023-06-23T01:31:50.953 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-06-23T01:31:50.961 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-06-23T01:31:50.961 DEBUG:teuthology.orchestra.run.smithi092:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:31:50.963 DEBUG:teuthology.orchestra.run.smithi119:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-06-23T01:31:51.004 INFO:teuthology.orchestra.run.smithi092.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:31:51.013 INFO:teuthology.orchestra.run.smithi119.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-06-23T01:31:51.206 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-06-23T01:31:51.206 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi092.front.sepia.ceph.com 2023-06-23T01:31:51.207 DEBUG:teuthology.orchestra.run.smithi092:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:31:51.265 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi119.front.sepia.ceph.com 2023-06-23T01:31:51.266 DEBUG:teuthology.orchestra.run.smithi119:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-06-23T01:31:51.284 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-06-23T01:31:51.285 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:31:51.307 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-06-23T01:31:51.361 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-06-23T01:31:51.361 DEBUG:teuthology.orchestra.run.smithi092:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:31:51.384 DEBUG:teuthology.orchestra.run.smithi119:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-06-23T01:31:51.470 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-06-23T01:31:51.479 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-06-23T01:31:51.479 DEBUG:teuthology.orchestra.run.smithi092:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:31:51.504 DEBUG:teuthology.orchestra.run.smithi119:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-06-23T01:31:51.528 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-06-23T01:31:51.539 DEBUG:teuthology.orchestra.run.smithi092:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:31:51.546 DEBUG:teuthology.orchestra.run.smithi119:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:31:51.567 INFO:teuthology.orchestra.run.smithi092.stdout:kernel.core_pattern = core 2023-06-23T01:31:51.592 INFO:teuthology.orchestra.run.smithi119.stdout:kernel.core_pattern = core 2023-06-23T01:31:51.605 DEBUG:teuthology.orchestra.run.smithi092:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:31:51.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:31:51.635 DEBUG:teuthology.orchestra.run.smithi119:> test -e /home/ubuntu/cephtest/archive/coredump 2023-06-23T01:31:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-06-23T01:31:51.659 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-06-23T01:31:51.669 INFO:teuthology.task.internal:Transferring archived files... 2023-06-23T01:31:51.670 DEBUG:teuthology.misc:Transferring archived files from smithi092:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/remote/smithi092 2023-06-23T01:31:51.671 DEBUG:teuthology.orchestra.run.smithi092:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:31:51.709 DEBUG:teuthology.misc:Transferring archived files from smithi119:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967/remote/smithi119 2023-06-23T01:31:51.710 DEBUG:teuthology.orchestra.run.smithi119:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-06-23T01:31:51.748 INFO:teuthology.task.internal:Removing archive directory... 2023-06-23T01:31:51.749 DEBUG:teuthology.orchestra.run.smithi092:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:31:51.751 DEBUG:teuthology.orchestra.run.smithi119:> rm -rf -- /home/ubuntu/cephtest/archive 2023-06-23T01:31:51.792 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-06-23T01:31:51.803 INFO:teuthology.task.internal:Not uploading archives. 2023-06-23T01:31:51.803 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-06-23T01:31:51.813 INFO:teuthology.task.internal:Tidying up after the test... 2023-06-23T01:31:51.813 DEBUG:teuthology.orchestra.run.smithi092:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:31:51.815 DEBUG:teuthology.orchestra.run.smithi119:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-06-23T01:31:51.827 INFO:teuthology.orchestra.run.smithi092.stdout: 262153 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:31 /home/ubuntu/cephtest 2023-06-23T01:31:51.846 INFO:teuthology.orchestra.run.smithi119.stdout: 262149 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 23 01:31 /home/ubuntu/cephtest 2023-06-23T01:31:51.847 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-06-23T01:31:51.857 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-06-23T01:31:51.883 INFO:teuthology.nuke:Checking targets against current locks 2023-06-23T01:31:51.909 DEBUG:teuthology.nuke:shortname: smithi092 2023-06-23T01:31:51.910 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:31:51.935 DEBUG:teuthology.nuke:shortname: smithi119 2023-06-23T01:31:51.936 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-06-23T01:31:51.939 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi092.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 01:12:30.125762', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCyCtGo6+Kvz2SWXvHcKq7TwgALZE7O2ZdSSDCKapdOYVDM/+QueBF/SOjjHzoK5xE3qRycvt4MeThoeNOlqGjBDuBnQ1xGPKdvQk852T9BVxz4/2FXr4bBQkf3meLqHxBufyzi07l7SA2ix1Vt6cZKib1QD9uAQL2chWtiTBItmBT16F2fP5qTm46PP+qRVTce+SWbH6QzOXCXiW8Td4zJurxt7VUlWBOVGh8dndG5QnoBLSndUMyiRI+dz34yhUPtRiWXhpDFQXO0Fx3psirsHTyRLczUunFKD3/TpRhcztV4LefNT22EheSHzUazFylz0kl0G9kf4KerisAgZPlJnV9288oXEQDZsgJ3MdZYX9YYcH66DFaveR2jfFYXTBlZ88DEVlQX8o+ThLVFc+K/UwNVlkzBNfsXrkLt+2fYXSgOem9cEuVRue/aG1b4e6Z/Gs2c4UQrwPTk3qA1M8RgQaWMDxq8xtrzs3upVTXBbHJrdjGjbZuVj+/jOeXVyZ8='} 2023-06-23T01:31:51.961 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi119.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-06-22_15:25:25-fs-wip-yuri4-testing-2023-06-21-0846-quincy-distro-default-smithi/7311967', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.4', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-06-23 01:12:30.127509', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbatMXeLcMLuQDmdjxHrNuwug4CM7BnUzrhJ0MzSXJQPq5T+21eDfm3gFGTmd8j1sLTQGFLEZdVxC5LMsbJg4z5BLpBBtuwaFKNxM54ePZPOFwXTQx8uUC8aY/xJEecFwDcaIYjlzuQF7zedT+8ZQLWWoUOKmyOqq/VHZIyU0LXNy1yWR3ZSX0Y+L0AjXXlRB85AwxoXlR0mIwXmeT39ozkK/jqz+FDtlnh6mVAf9zpsdl8jKXcMRFKU8uAE4ByOfSiQFkkKIt5FXkGFkc55IvLuodTRi7eB/h08ZeL+ApFKjeOP9WZjNB+mMfLHKBwTBoNl3hYaQZWGPAShxgfeG0tqysLYKVcRfAleAvafOpSWBoaf7JLuhAcZDv1eshFfC98fkIJFlahh0xmzyzwFKXk3zlwmzn+EmOhpUCiYIsksb+6gSz78RsQjAZMFrZukMct8DTaJwqh+gujOG1LEvh4r1jiIvQGokgwAQ9DYQ5ULPE2MqdkpEe9hW7x7npks0='} 2023-06-23T01:31:51.996 INFO:teuthology.orchestra.console:Power off smithi092 2023-06-23T01:31:51.998 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:31:52.025 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:31:52.045 INFO:teuthology.orchestra.console:Power off smithi119 2023-06-23T01:31:52.046 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-06-23T01:31:52.069 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:31:56.058 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:31:56.093 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:32:00.174 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi092.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:32:00.206 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi119.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-06-23T01:32:00.387 INFO:teuthology.orchestra.console:Power off for smithi092 completed 2023-06-23T01:32:00.421 INFO:teuthology.orchestra.console:Power off for smithi119 completed 2023-06-23T01:32:00.526 INFO:teuthology.run:Summary data: description: fs/volumes/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{ubuntu_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/volumes/{overrides test/basic}} duration: 694.2578630447388 failure_reason: 'Command failed on smithi092 with status 22: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fa8f36015b58456581e8a1ae62ba5836 status: fail success: false 2023-06-23T01:32:00.527 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-06-23T01:32:00.623 INFO:teuthology.run:FAIL