2023-03-25T00:47:03.058 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-25T00:47:03.059 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219658 --owner scheduled_pdonnell@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} -- /home/teuthworker/archive/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219658/orig.config.yaml 2023-03-25T00:47:03.103 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-25T00:47:03.182 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219658 branch: wip-pdonnell-testing-20230323.162417 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} email: pdonnell@redhat.com first_in_suite: false job_id: '7219658' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230323.162417 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: c19e8f2a497e95f75abc43ec14bc6095e30dd2cb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c19e8f2a497e95f75abc43ec14bc6095e30dd2cb kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230323.162417 sha1: 1e7de2f05230c7419acf5ca7f5fd22964e30ab77 owner: scheduled_pdonnell@teuthology priority: 55 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 sha1: c19e8f2a497e95f75abc43ec14bc6095e30dd2cb sleep_before_teardown: 0 suite: fs suite_branch: wip-pdonnell-testing-20230323.162417 suite_path: /home/teuthworker/src/github.com_batrick_ceph_1e7de2f05230c7419acf5ca7f5fd22964e30ab77/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 1e7de2f05230c7419acf5ca7f5fd22964e30ab77 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi115.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= smithi121.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: parallel-gzip teuthology_sha1: fe9b6e52013911547168fef4a8506325b6c76139 timestamp: 2023-03-24_23:11:30 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4538 2023-03-25T00:47:03.183 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_1e7de2f05230c7419acf5ca7f5fd22964e30ab77/qa; will attempt to use it 2023-03-25T00:47:03.184 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_1e7de2f05230c7419acf5ca7f5fd22964e30ab77/qa/tasks 2023-03-25T00:47:03.184 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-03-25T00:47:03.185 INFO:teuthology.task.internal:Checking packages... 2023-03-25T00:47:03.214 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb' 2023-03-25T00:47:03.215 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-03-25T00:47:03.215 INFO:teuthology.packaging:ref: None 2023-03-25T00:47:03.215 INFO:teuthology.packaging:tag: None 2023-03-25T00:47:03.216 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230323.162417 2023-03-25T00:47:03.216 INFO:teuthology.packaging:sha1: c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-25T00:47:03.216 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230323.162417 2023-03-25T00:47:03.359 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3124.gc19e8f2a 2023-03-25T00:47:03.359 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-03-25T00:47:03.368 INFO:teuthology.task.internal:no buildpackages task found 2023-03-25T00:47:03.368 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-03-25T00:47:03.387 INFO:teuthology.task.internal:Saving configuration 2023-03-25T00:47:03.404 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-03-25T00:47:03.412 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-25T00:47:03.441 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi115.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219658', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-25 00:41:55.936928', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-25T00:47:03.467 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219658', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-25 00:41:55.938602', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-25T00:47:03.468 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-03-25T00:47:03.476 INFO:teuthology.task.internal:roles: ubuntu@smithi115.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-25T00:47:03.476 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-25T00:47:03.476 INFO:teuthology.run_tasks:Running task console_log... 2023-03-25T00:47:03.563 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa812c8d5e0>, signals=[15]) 2023-03-25T00:47:03.565 INFO:teuthology.run_tasks:Running task internal.connect... 2023-03-25T00:47:03.572 INFO:teuthology.task.internal:Opening connections... 2023-03-25T00:47:03.572 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi115.front.sepia.ceph.com 2023-03-25T00:47:03.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-25T00:47:03.640 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2023-03-25T00:47:03.642 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-25T00:47:03.710 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-03-25T00:47:03.719 DEBUG:teuthology.orchestra.run.smithi115:> uname -m 2023-03-25T00:47:03.739 INFO:teuthology.orchestra.run.smithi115.stdout:x86_64 2023-03-25T00:47:03.739 DEBUG:teuthology.orchestra.run.smithi115:> cat /etc/os-release 2023-03-25T00:47:03.798 INFO:teuthology.orchestra.run.smithi115.stdout:NAME="Red Hat Enterprise Linux" 2023-03-25T00:47:03.798 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION="8.6 (Ootpa)" 2023-03-25T00:47:03.799 INFO:teuthology.orchestra.run.smithi115.stdout:ID="rhel" 2023-03-25T00:47:03.799 INFO:teuthology.orchestra.run.smithi115.stdout:ID_LIKE="fedora" 2023-03-25T00:47:03.799 INFO:teuthology.orchestra.run.smithi115.stdout:VERSION_ID="8.6" 2023-03-25T00:47:03.799 INFO:teuthology.orchestra.run.smithi115.stdout:PLATFORM_ID="platform:el8" 2023-03-25T00:47:03.799 INFO:teuthology.orchestra.run.smithi115.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-25T00:47:03.799 INFO:teuthology.orchestra.run.smithi115.stdout:ANSI_COLOR="0;31" 2023-03-25T00:47:03.799 INFO:teuthology.orchestra.run.smithi115.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-25T00:47:03.800 INFO:teuthology.orchestra.run.smithi115.stdout:HOME_URL="https://www.redhat.com/" 2023-03-25T00:47:03.800 INFO:teuthology.orchestra.run.smithi115.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-25T00:47:03.800 INFO:teuthology.orchestra.run.smithi115.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-25T00:47:03.800 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:47:03.800 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-25T00:47:03.800 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-25T00:47:03.800 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-25T00:47:03.801 INFO:teuthology.orchestra.run.smithi115.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-25T00:47:03.802 INFO:teuthology.lock.ops:Updating smithi115.front.sepia.ceph.com on lock server 2023-03-25T00:47:03.824 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2023-03-25T00:47:03.844 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2023-03-25T00:47:03.844 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2023-03-25T00:47:03.903 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="Red Hat Enterprise Linux" 2023-03-25T00:47:03.903 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8.6 (Ootpa)" 2023-03-25T00:47:03.903 INFO:teuthology.orchestra.run.smithi121.stdout:ID="rhel" 2023-03-25T00:47:03.903 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="fedora" 2023-03-25T00:47:03.904 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8.6" 2023-03-25T00:47:03.904 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2023-03-25T00:47:03.904 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-25T00:47:03.904 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2023-03-25T00:47:03.904 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-25T00:47:03.904 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://www.redhat.com/" 2023-03-25T00:47:03.904 INFO:teuthology.orchestra.run.smithi121.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-25T00:47:03.904 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-25T00:47:03.905 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:47:03.905 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-25T00:47:03.905 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-25T00:47:03.905 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-25T00:47:03.905 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-25T00:47:03.905 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2023-03-25T00:47:03.932 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-03-25T00:47:03.942 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-03-25T00:47:03.950 INFO:teuthology.task.internal:Checking for old test directory... 2023-03-25T00:47:03.951 DEBUG:teuthology.orchestra.run.smithi115:> test '!' -e /home/ubuntu/cephtest 2023-03-25T00:47:03.953 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2023-03-25T00:47:03.971 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-03-25T00:47:03.979 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-03-25T00:47:03.979 DEBUG:teuthology.orchestra.run.smithi115:> test -z $(ls -A /var/lib/ceph) 2023-03-25T00:47:04.013 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2023-03-25T00:47:04.050 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-03-25T00:47:04.114 INFO:teuthology.run_tasks:Running task kernel... 2023-03-25T00:47:04.131 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-03-25T00:47:04.132 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-03-25T00:47:04.132 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-03-25T00:47:04.133 DEBUG:teuthology.orchestra.run.smithi115:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-25T00:47:04.134 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-25T00:47:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-25T00:47:04.152 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2023-03-25T00:47:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-25T00:47:04.154 DEBUG:teuthology.orchestra.run.smithi115:> uname -r 2023-03-25T00:47:04.212 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-372.9.1.el8.x86_64 2023-03-25T00:47:04.212 INFO:teuthology.task.kernel:Running kernel on smithi121: 4.18.0-372.9.1.el8.x86_64 2023-03-25T00:47:04.213 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2023-03-25T00:47:04.214 INFO:teuthology.orchestra.run.smithi115.stdout:4.18.0-372.9.1.el8.x86_64 2023-03-25T00:47:04.215 INFO:teuthology.task.kernel:Running kernel on smithi115: 4.18.0-372.9.1.el8.x86_64 2023-03-25T00:47:04.215 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum install -y kernel 2023-03-25T00:47:05.391 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-03-25T00:47:05.392 INFO:teuthology.orchestra.run.smithi121.stdout:Unable to read consumer identity 2023-03-25T00:47:05.416 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-03-25T00:47:05.417 INFO:teuthology.orchestra.run.smithi115.stdout:Unable to read consumer identity 2023-03-25T00:47:06.822 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-03-25T00:47:06.862 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-03-25T00:47:08.287 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 11 MB/s | 14 MB 00:01 2023-03-25T00:47:08.731 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 7.3 MB/s | 14 MB 00:01 2023-03-25T00:47:11.363 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 364 kB/s | 24 kB 00:00 2023-03-25T00:47:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 292 kB/s | 24 kB 00:00 2023-03-25T00:47:12.015 INFO:teuthology.orchestra.run.smithi115.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-03-25T00:47:12.062 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-03-25T00:47:12.063 INFO:teuthology.orchestra.run.smithi115.stdout:Nothing to do. 2023-03-25T00:47:12.063 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-03-25T00:47:12.185 DEBUG:teuthology.orchestra.run.smithi115:> echo no | sudo yum reinstall kernel || true 2023-03-25T00:47:12.442 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-03-25T00:47:12.492 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-03-25T00:47:12.493 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2023-03-25T00:47:12.494 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-03-25T00:47:12.516 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-03-25T00:47:12.516 INFO:teuthology.orchestra.run.smithi115.stdout:Unable to read consumer identity 2023-03-25T00:47:12.613 DEBUG:teuthology.orchestra.run.smithi121:> echo no | sudo yum reinstall kernel || true 2023-03-25T00:47:12.770 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:01 ago on Sat 25 Mar 2023 12:47:11 AM UTC. 2023-03-25T00:47:12.813 INFO:teuthology.orchestra.run.smithi115.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-25T00:47:12.856 INFO:teuthology.orchestra.run.smithi115.stderr:Error: No packages marked for reinstall. 2023-03-25T00:47:12.901 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum reinstall -y kernel || true 2023-03-25T00:47:12.941 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-03-25T00:47:12.942 INFO:teuthology.orchestra.run.smithi121.stdout:Unable to read consumer identity 2023-03-25T00:47:13.200 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:02 ago on Sat 25 Mar 2023 12:47:11 AM UTC. 2023-03-25T00:47:13.230 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-03-25T00:47:13.231 INFO:teuthology.orchestra.run.smithi115.stdout:Unable to read consumer identity 2023-03-25T00:47:13.240 INFO:teuthology.orchestra.run.smithi121.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-25T00:47:13.285 INFO:teuthology.orchestra.run.smithi121.stderr:Error: No packages marked for reinstall. 2023-03-25T00:47:13.332 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum reinstall -y kernel || true 2023-03-25T00:47:13.483 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:00:02 ago on Sat 25 Mar 2023 12:47:11 AM UTC. 2023-03-25T00:47:13.523 INFO:teuthology.orchestra.run.smithi115.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-25T00:47:13.546 INFO:teuthology.orchestra.run.smithi115.stderr:Error: No packages marked for reinstall. 2023-03-25T00:47:13.590 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q kernel --last | head -n 1 2023-03-25T00:47:13.660 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-03-25T00:47:13.661 INFO:teuthology.orchestra.run.smithi121.stdout:Unable to read consumer identity 2023-03-25T00:47:13.669 INFO:teuthology.orchestra.run.smithi115.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-03-25T00:47:13.671 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-03-25T00:47:13.671 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-03-25T00:47:13.672 DEBUG:teuthology.parallel:result is None 2023-03-25T00:47:13.909 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:02 ago on Sat 25 Mar 2023 12:47:11 AM UTC. 2023-03-25T00:47:13.949 INFO:teuthology.orchestra.run.smithi121.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-25T00:47:13.967 INFO:teuthology.orchestra.run.smithi121.stderr:Error: No packages marked for reinstall. 2023-03-25T00:47:14.013 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel --last | head -n 1 2023-03-25T00:47:14.092 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-03-25T00:47:14.094 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-03-25T00:47:14.094 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-03-25T00:47:14.094 DEBUG:teuthology.parallel:result is None 2023-03-25T00:47:14.095 INFO:teuthology.run_tasks:Running task internal.base... 2023-03-25T00:47:14.102 INFO:teuthology.task.internal:Creating test directory... 2023-03-25T00:47:14.102 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-25T00:47:14.104 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-25T00:47:14.121 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-03-25T00:47:14.129 INFO:teuthology.run_tasks:Running task internal.archive... 2023-03-25T00:47:14.138 INFO:teuthology.task.internal:Creating archive directory... 2023-03-25T00:47:14.138 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-25T00:47:14.161 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-25T00:47:14.184 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-03-25T00:47:14.192 INFO:teuthology.task.internal:Enabling coredump saving... 2023-03-25T00:47:14.193 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-25T00:47:14.222 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-25T00:47:14.261 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-25T00:47:14.265 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-25T00:47:14.272 INFO:teuthology.orchestra.run.smithi115.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-25T00:47:14.275 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-25T00:47:14.277 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-03-25T00:47:14.284 INFO:teuthology.task.internal:Configuring sudo... 2023-03-25T00:47:14.285 DEBUG:teuthology.orchestra.run.smithi115:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-25T00:47:14.316 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-25T00:47:14.343 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-03-25T00:47:14.353 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-03-25T00:47:14.354 DEBUG:teuthology.orchestra.run.smithi115:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-25T00:47:14.383 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-25T00:47:14.399 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-25T00:47:14.485 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-25T00:47:14.537 DEBUG:teuthology.orchestra.run.smithi115:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-25T00:47:14.623 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-25T00:47:14.661 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:47:14.662 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-25T00:47:14.728 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-25T00:47:14.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-25T00:47:14.818 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-25T00:47:14.905 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-25T00:47:14.943 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:47:14.943 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-25T00:47:15.010 DEBUG:teuthology.orchestra.run.smithi115:> sudo service rsyslog restart 2023-03-25T00:47:15.012 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2023-03-25T00:47:15.064 INFO:teuthology.orchestra.run.smithi115.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-25T00:47:15.117 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-25T00:47:15.468 INFO:teuthology.run_tasks:Running task internal.timer... 2023-03-25T00:47:15.476 INFO:teuthology.task.internal:Starting timer... 2023-03-25T00:47:15.476 INFO:teuthology.run_tasks:Running task pcp... 2023-03-25T00:47:15.489 INFO:teuthology.run_tasks:Running task selinux... 2023-03-25T00:47:15.499 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-03-25T00:47:15.499 DEBUG:teuthology.orchestra.run.smithi115:> sudo service auditd rotate 2023-03-25T00:47:15.574 INFO:teuthology.orchestra.run.smithi115.stdout:Rotating logs: 2023-03-25T00:47:15.576 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2023-03-25T00:47:15.638 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2023-03-25T00:47:15.639 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-03-25T00:47:15.640 DEBUG:teuthology.orchestra.run.smithi115:> /usr/sbin/getenforce 2023-03-25T00:47:15.658 INFO:teuthology.orchestra.run.smithi115.stdout:Permissive 2023-03-25T00:47:15.659 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2023-03-25T00:47:15.684 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2023-03-25T00:47:15.684 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi115.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi121.front.sepia.ceph.com': 'permissive'} 2023-03-25T00:47:15.685 DEBUG:teuthology.orchestra.run.smithi115:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-25T00:47:15.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-25T00:47:15.725 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-25T00:47:15.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-25T00:47:15.753 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-03-25T00:47:15.754 DEBUG:teuthology.orchestra.run.smithi115:> sudo /usr/sbin/setenforce permissive 2023-03-25T00:47:15.792 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2023-03-25T00:47:15.821 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-03-25T00:47:15.830 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-03-25T00:47:15.836 INFO:teuthology.repo_utils:Fetching main from origin 2023-03-25T00:47:15.963 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-03-25T00:47:15.986 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-03-25T00:47:15.987 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi115.front.sepia.ceph.com,smithi121.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-03-25T00:54:34.013 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi115.front.sepia.ceph.com'), Remote(name='ubuntu@smithi121.front.sepia.ceph.com')] 2023-03-25T00:54:34.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi115.front.sepia.ceph.com' 2023-03-25T00:54:34.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi115.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-25T00:54:34.084 DEBUG:teuthology.orchestra.run.smithi115:> true 2023-03-25T00:54:34.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2023-03-25T00:54:34.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-25T00:54:34.220 DEBUG:teuthology.orchestra.run.smithi121:> true 2023-03-25T00:54:34.287 INFO:teuthology.run_tasks:Running task clock... 2023-03-25T00:54:34.302 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-03-25T00:54:34.302 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-25T00:54:34.303 DEBUG:teuthology.orchestra.run.smithi115:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-25T00:54:34.305 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-25T00:54:34.306 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-25T00:54:34.336 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-25T00:54:34.351 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-25T00:54:34.360 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-25T00:54:34.375 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-25T00:54:34.416 INFO:teuthology.orchestra.run.smithi115.stderr:sudo: ntpd: command not found 2023-03-25T00:54:34.427 INFO:teuthology.orchestra.run.smithi115.stdout:506 Cannot talk to daemon 2023-03-25T00:54:34.442 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-25T00:54:34.446 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2023-03-25T00:54:34.456 INFO:teuthology.orchestra.run.smithi115.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-25T00:54:34.457 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2023-03-25T00:54:34.471 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-25T00:54:34.486 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-25T00:54:34.525 INFO:teuthology.orchestra.run.smithi115.stderr:bash: ntpq: command not found 2023-03-25T00:54:34.528 INFO:teuthology.orchestra.run.smithi115.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-25T00:54:34.528 INFO:teuthology.orchestra.run.smithi115.stdout:=============================================================================== 2023-03-25T00:54:34.529 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-25T00:54:34.529 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-25T00:54:34.529 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-25T00:54:34.530 INFO:teuthology.orchestra.run.smithi115.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-25T00:54:34.564 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2023-03-25T00:54:34.568 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-25T00:54:34.568 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2023-03-25T00:54:34.568 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-25T00:54:34.569 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-25T00:54:34.569 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-25T00:54:34.569 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-25T00:54:34.570 INFO:teuthology.run_tasks:Running task install... 2023-03-25T00:54:34.578 DEBUG:teuthology.task.install:project ceph 2023-03-25T00:54:34.578 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'}} 2023-03-25T00:54:34.579 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'} 2023-03-25T00:54:34.579 INFO:teuthology.task.install:Using flavor: default 2023-03-25T00:54:34.584 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-03-25T00:54:34.585 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-03-25T00:54:34.585 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb', 'tag': None, 'wait_for_package': False} 2023-03-25T00:54:34.586 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-25T00:54:34.588 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb', 'tag': None, 'wait_for_package': False} 2023-03-25T00:54:34.589 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-25T00:54:34.748 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/ 2023-03-25T00:54:34.749 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3124.gc19e8f2a 2023-03-25T00:54:34.762 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/ 2023-03-25T00:54:34.762 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3124.gc19e8f2a 2023-03-25T00:54:34.862 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-25T00:54:34.862 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:54:34.863 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-25T00:54:34.867 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-25T00:54:34.868 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:54:34.868 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-25T00:54:34.890 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-25T00:54:34.890 DEBUG:teuthology.orchestra.run.smithi115:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-25T00:54:34.895 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-25T00:54:34.895 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-25T00:54:34.957 DEBUG:teuthology.orchestra.run.smithi115:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-25T00:54:34.962 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-25T00:54:35.034 DEBUG:teuthology.orchestra.run.smithi115:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-25T00:54:35.039 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-25T00:54:35.100 INFO:teuthology.orchestra.run.smithi115.stdout:check_obsoletes = 1 2023-03-25T00:54:35.101 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum clean all 2023-03-25T00:54:35.104 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2023-03-25T00:54:35.106 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2023-03-25T00:54:35.472 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-03-25T00:54:35.476 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-03-25T00:54:35.728 INFO:teuthology.orchestra.run.smithi115.stdout:44 files removed 2023-03-25T00:54:35.745 INFO:teuthology.orchestra.run.smithi121.stdout:44 files removed 2023-03-25T00:54:35.774 DEBUG:teuthology.orchestra.run.smithi115:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-25T00:54:35.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-25T00:54:36.105 INFO:teuthology.orchestra.run.smithi115.stdout:Updating Subscription Management repositories. 2023-03-25T00:54:36.124 INFO:teuthology.orchestra.run.smithi121.stdout:Updating Subscription Management repositories. 2023-03-25T00:54:36.709 INFO:teuthology.orchestra.run.smithi115.stdout:ceph packages for x86_64 313 kB/s | 80 kB 00:00 2023-03-25T00:54:36.727 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 311 kB/s | 80 kB 00:00 2023-03-25T00:54:36.976 INFO:teuthology.orchestra.run.smithi115.stdout:ceph noarch packages 40 kB/s | 9.4 kB 00:00 2023-03-25T00:54:36.986 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 41 kB/s | 9.4 kB 00:00 2023-03-25T00:54:37.237 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 6.5 kB/s | 1.5 kB 00:00 2023-03-25T00:54:37.243 INFO:teuthology.orchestra.run.smithi115.stdout:ceph source packages 6.0 kB/s | 1.5 kB 00:00 2023-03-25T00:54:37.402 INFO:teuthology.orchestra.run.smithi115.stdout:Copr repo for python3-asyncssh owned by ceph 25 kB/s | 3.5 kB 00:00 2023-03-25T00:54:37.404 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-03-25T00:54:38.580 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 12 MB/s | 14 MB 00:01 2023-03-25T00:54:38.594 INFO:teuthology.orchestra.run.smithi115.stdout:Extra Packages for Enterprise Linux 12 MB/s | 14 MB 00:01 2023-03-25T00:54:41.663 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 415 kB/s | 24 kB 00:00 2023-03-25T00:54:41.686 INFO:teuthology.orchestra.run.smithi115.stdout:lab-extras 437 kB/s | 24 kB 00:00 2023-03-25T00:54:42.581 INFO:teuthology.orchestra.run.smithi121.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 62 MB/s | 53 MB 00:00 2023-03-25T00:54:42.621 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 55 MB/s | 47 MB 00:00 2023-03-25T00:54:50.563 INFO:teuthology.orchestra.run.smithi115.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 61 MB/s | 53 MB 00:00 2023-03-25T00:54:50.573 INFO:teuthology.orchestra.run.smithi121.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 60 MB/s | 47 MB 00:00 2023-03-25T00:55:01.470 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-25T00:55:01.471 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-25T00:55:01.486 INFO:teuthology.orchestra.run.smithi121.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-25T00:55:01.487 INFO:teuthology.orchestra.run.smithi121.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-25T00:55:01.488 INFO:teuthology.orchestra.run.smithi121.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-25T00:55:01.489 INFO:teuthology.orchestra.run.smithi121.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-25T00:55:01.491 INFO:teuthology.orchestra.run.smithi121.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-25T00:55:01.494 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-25T00:55:01.495 INFO:teuthology.orchestra.run.smithi121.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-25T00:55:01.514 INFO:teuthology.orchestra.run.smithi115.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-25T00:55:01.515 INFO:teuthology.orchestra.run.smithi115.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-25T00:55:01.530 INFO:teuthology.orchestra.run.smithi115.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-25T00:55:01.531 INFO:teuthology.orchestra.run.smithi115.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-25T00:55:01.532 INFO:teuthology.orchestra.run.smithi115.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-25T00:55:01.533 INFO:teuthology.orchestra.run.smithi115.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-25T00:55:01.536 INFO:teuthology.orchestra.run.smithi115.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-25T00:55:01.540 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-25T00:55:01.540 INFO:teuthology.orchestra.run.smithi115.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-25T00:55:01.627 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2023-03-25T00:55:01.639 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-03-25T00:55:01.639 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2023-03-25T00:55:01.639 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-03-25T00:55:01.639 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2023-03-25T00:55:01.639 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-25T00:55:01.640 INFO:teuthology.orchestra.run.smithi121.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-25T00:55:01.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 6.4 k 2023-03-25T00:55:01.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 5.4 M 2023-03-25T00:55:01.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 846 k 2023-03-25T00:55:01.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 165 k 2023-03-25T00:55:01.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 1.5 M 2023-03-25T00:55:01.640 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 123 k 2023-03-25T00:55:01.641 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 1.7 M 2023-03-25T00:55:01.641 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 7.4 M 2023-03-25T00:55:01.641 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 47 k 2023-03-25T00:55:01.641 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 13 M 2023-03-25T00:55:01.641 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 41 M 2023-03-25T00:55:01.641 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 263 k 2023-03-25T00:55:01.641 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 83 k 2023-03-25T00:55:01.641 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 258 k 2023-03-25T00:55:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 18 k 2023-03-25T00:55:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-25T00:55:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-25T00:55:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-25T00:55:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 30 k 2023-03-25T00:55:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 769 k 2023-03-25T00:55:01.642 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 139 k 2023-03-25T00:55:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-25T00:55:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-25T00:55:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-25T00:55:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-25T00:55:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-25T00:55:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-25T00:55:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 190 k 2023-03-25T00:55:01.643 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 370 k 2023-03-25T00:55:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 364 k 2023-03-25T00:55:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 107 k 2023-03-25T00:55:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 94 k 2023-03-25T00:55:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.4 M 2023-03-25T00:55:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 179 k 2023-03-25T00:55:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-25T00:55:01.644 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2023-03-25T00:55:01.644 INFO:teuthology.orchestra.run.smithi121.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-25T00:55:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.7 M 2023-03-25T00:55:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.6 M 2023-03-25T00:55:01.645 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2023-03-25T00:55:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-25T00:55:01.645 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 22 M 2023-03-25T00:55:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 23 k 2023-03-25T00:55:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 2.3 M 2023-03-25T00:55:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 239 k 2023-03-25T00:55:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 4.6 M 2023-03-25T00:55:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 18 M 2023-03-25T00:55:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 15 k 2023-03-25T00:55:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 24 k 2023-03-25T00:55:01.646 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-25T00:55:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-25T00:55:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-25T00:55:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-25T00:55:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-25T00:55:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 137 k 2023-03-25T00:55:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-25T00:55:01.647 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-25T00:55:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-25T00:55:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-25T00:55:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-25T00:55:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-25T00:55:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 474 k 2023-03-25T00:55:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-25T00:55:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 6.5 M 2023-03-25T00:55:01.648 INFO:teuthology.orchestra.run.smithi121.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-25T00:55:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-25T00:55:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-25T00:55:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-25T00:55:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-25T00:55:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-25T00:55:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-25T00:55:01.649 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-25T00:55:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-25T00:55:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-25T00:55:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-25T00:55:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-25T00:55:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-25T00:55:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-25T00:55:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-25T00:55:01.650 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-25T00:55:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 45 k 2023-03-25T00:55:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 117 k 2023-03-25T00:55:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-25T00:55:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-25T00:55:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-25T00:55:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-25T00:55:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-25T00:55:01.651 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-25T00:55:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-25T00:55:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-25T00:55:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-25T00:55:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-25T00:55:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-25T00:55:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-25T00:55:01.652 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-25T00:55:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-25T00:55:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-25T00:55:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-25T00:55:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-25T00:55:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-25T00:55:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-25T00:55:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-25T00:55:01.653 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-25T00:55:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-25T00:55:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-25T00:55:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-25T00:55:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-25T00:55:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-25T00:55:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-25T00:55:01.654 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-25T00:55:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-25T00:55:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-25T00:55:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-25T00:55:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-25T00:55:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-25T00:55:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-25T00:55:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-25T00:55:01.655 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-25T00:55:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-25T00:55:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-25T00:55:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-25T00:55:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-25T00:55:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-25T00:55:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-25T00:55:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-25T00:55:01.656 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-25T00:55:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-25T00:55:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-25T00:55:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-25T00:55:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-25T00:55:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-25T00:55:01.657 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling module streams: 2023-03-25T00:55:01.657 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql 10 2023-03-25T00:55:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:55:01.658 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2023-03-25T00:55:01.658 INFO:teuthology.orchestra.run.smithi121.stdout:=========================================================================================================================== 2023-03-25T00:55:01.658 INFO:teuthology.orchestra.run.smithi121.stdout:Install 128 Packages 2023-03-25T00:55:01.658 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 3 Packages 2023-03-25T00:55:01.658 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:55:01.659 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 197 M 2023-03-25T00:55:01.659 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2023-03-25T00:55:01.669 INFO:teuthology.orchestra.run.smithi115.stdout:Dependencies resolved. 2023-03-25T00:55:01.681 INFO:teuthology.orchestra.run.smithi115.stdout:=========================================================================================================================== 2023-03-25T00:55:01.682 INFO:teuthology.orchestra.run.smithi115.stdout: Package Arch Version Repository Size 2023-03-25T00:55:01.682 INFO:teuthology.orchestra.run.smithi115.stdout:=========================================================================================================================== 2023-03-25T00:55:01.682 INFO:teuthology.orchestra.run.smithi115.stdout:Installing: 2023-03-25T00:55:01.682 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-25T00:55:01.682 INFO:teuthology.orchestra.run.smithi115.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-25T00:55:01.682 INFO:teuthology.orchestra.run.smithi115.stdout: ceph x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 6.4 k 2023-03-25T00:55:01.682 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 5.4 M 2023-03-25T00:55:01.683 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 846 k 2023-03-25T00:55:01.683 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 165 k 2023-03-25T00:55:01.683 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 1.5 M 2023-03-25T00:55:01.683 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 123 k 2023-03-25T00:55:01.683 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 1.7 M 2023-03-25T00:55:01.683 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 7.4 M 2023-03-25T00:55:01.683 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 47 k 2023-03-25T00:55:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 13 M 2023-03-25T00:55:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 41 M 2023-03-25T00:55:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 263 k 2023-03-25T00:55:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 83 k 2023-03-25T00:55:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 258 k 2023-03-25T00:55:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 18 k 2023-03-25T00:55:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-25T00:55:01.684 INFO:teuthology.orchestra.run.smithi115.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-25T00:55:01.685 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-25T00:55:01.685 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 30 k 2023-03-25T00:55:01.685 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 769 k 2023-03-25T00:55:01.685 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 139 k 2023-03-25T00:55:01.685 INFO:teuthology.orchestra.run.smithi115.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-25T00:55:01.685 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-25T00:55:01.685 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-25T00:55:01.686 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-25T00:55:01.686 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-25T00:55:01.686 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-25T00:55:01.686 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 190 k 2023-03-25T00:55:01.686 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 370 k 2023-03-25T00:55:01.687 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 364 k 2023-03-25T00:55:01.687 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 107 k 2023-03-25T00:55:01.687 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 94 k 2023-03-25T00:55:01.687 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.4 M 2023-03-25T00:55:01.687 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 179 k 2023-03-25T00:55:01.687 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-25T00:55:01.687 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrading: 2023-03-25T00:55:01.688 INFO:teuthology.orchestra.run.smithi115.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-25T00:55:01.688 INFO:teuthology.orchestra.run.smithi115.stdout: librados2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.7 M 2023-03-25T00:55:01.688 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.6 M 2023-03-25T00:55:01.688 INFO:teuthology.orchestra.run.smithi115.stdout:Installing dependencies: 2023-03-25T00:55:01.688 INFO:teuthology.orchestra.run.smithi115.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-25T00:55:01.688 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 22 M 2023-03-25T00:55:01.688 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 23 k 2023-03-25T00:55:01.689 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 2.3 M 2023-03-25T00:55:01.689 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 239 k 2023-03-25T00:55:01.689 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 4.6 M 2023-03-25T00:55:01.689 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 18 M 2023-03-25T00:55:01.689 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 15 k 2023-03-25T00:55:01.689 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 24 k 2023-03-25T00:55:01.689 INFO:teuthology.orchestra.run.smithi115.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-25T00:55:01.689 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-25T00:55:01.690 INFO:teuthology.orchestra.run.smithi115.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-25T00:55:01.690 INFO:teuthology.orchestra.run.smithi115.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-25T00:55:01.690 INFO:teuthology.orchestra.run.smithi115.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-25T00:55:01.690 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 137 k 2023-03-25T00:55:01.690 INFO:teuthology.orchestra.run.smithi115.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-25T00:55:01.690 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-25T00:55:01.690 INFO:teuthology.orchestra.run.smithi115.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-25T00:55:01.690 INFO:teuthology.orchestra.run.smithi115.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-25T00:55:01.691 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-25T00:55:01.691 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-25T00:55:01.691 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 474 k 2023-03-25T00:55:01.691 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-25T00:55:01.691 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 6.5 M 2023-03-25T00:55:01.691 INFO:teuthology.orchestra.run.smithi115.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-25T00:55:01.691 INFO:teuthology.orchestra.run.smithi115.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-25T00:55:01.692 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-25T00:55:01.692 INFO:teuthology.orchestra.run.smithi115.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-25T00:55:01.692 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-25T00:55:01.692 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-25T00:55:01.692 INFO:teuthology.orchestra.run.smithi115.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-25T00:55:01.692 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-25T00:55:01.692 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-25T00:55:01.692 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-25T00:55:01.693 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-25T00:55:01.693 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-25T00:55:01.693 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-25T00:55:01.693 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-25T00:55:01.693 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-25T00:55:01.693 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-25T00:55:01.693 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 45 k 2023-03-25T00:55:01.693 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 117 k 2023-03-25T00:55:01.694 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-25T00:55:01.694 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-25T00:55:01.694 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-25T00:55:01.694 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-25T00:55:01.694 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-25T00:55:01.694 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-25T00:55:01.694 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-25T00:55:01.695 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-25T00:55:01.695 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-25T00:55:01.695 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-25T00:55:01.695 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-25T00:55:01.695 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-25T00:55:01.695 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-25T00:55:01.695 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-25T00:55:01.695 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-25T00:55:01.696 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-25T00:55:01.696 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-25T00:55:01.696 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-25T00:55:01.696 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-25T00:55:01.696 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-25T00:55:01.696 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-25T00:55:01.696 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-25T00:55:01.696 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-25T00:55:01.697 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-25T00:55:01.697 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-25T00:55:01.697 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-25T00:55:01.697 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-25T00:55:01.697 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-25T00:55:01.697 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-25T00:55:01.697 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-25T00:55:01.698 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-25T00:55:01.698 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-25T00:55:01.698 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-25T00:55:01.698 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-25T00:55:01.698 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-25T00:55:01.698 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-25T00:55:01.698 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-25T00:55:01.698 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-25T00:55:01.699 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-25T00:55:01.699 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-25T00:55:01.699 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-25T00:55:01.699 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-25T00:55:01.699 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-25T00:55:01.699 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-25T00:55:01.699 INFO:teuthology.orchestra.run.smithi115.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-25T00:55:01.699 INFO:teuthology.orchestra.run.smithi115.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-25T00:55:01.700 INFO:teuthology.orchestra.run.smithi115.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-25T00:55:01.700 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-25T00:55:01.700 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-25T00:55:01.700 INFO:teuthology.orchestra.run.smithi115.stdout:Enabling module streams: 2023-03-25T00:55:01.700 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql 10 2023-03-25T00:55:01.700 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:55:01.700 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction Summary 2023-03-25T00:55:01.701 INFO:teuthology.orchestra.run.smithi115.stdout:=========================================================================================================================== 2023-03-25T00:55:01.701 INFO:teuthology.orchestra.run.smithi115.stdout:Install 128 Packages 2023-03-25T00:55:01.701 INFO:teuthology.orchestra.run.smithi115.stdout:Upgrade 3 Packages 2023-03-25T00:55:01.701 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:55:01.701 INFO:teuthology.orchestra.run.smithi115.stdout:Total download size: 197 M 2023-03-25T00:55:01.702 INFO:teuthology.orchestra.run.smithi115.stdout:Downloading Packages: 2023-03-25T00:55:01.772 INFO:teuthology.orchestra.run.smithi121.stdout:(1/131): ceph-18.0.0-3124.gc19e8f2a.el8.x86_64. 54 kB/s | 6.4 kB 00:00 2023-03-25T00:55:01.829 INFO:teuthology.orchestra.run.smithi115.stdout:(1/131): ceph-18.0.0-3124.gc19e8f2a.el8.x86_64. 49 kB/s | 6.4 kB 00:00 2023-03-25T00:55:02.123 INFO:teuthology.orchestra.run.smithi121.stdout:(2/131): ceph-fuse-18.0.0-3124.gc19e8f2a.el8.x8 2.4 MB/s | 846 kB 00:00 2023-03-25T00:55:02.229 INFO:teuthology.orchestra.run.smithi115.stdout:(2/131): ceph-fuse-18.0.0-3124.gc19e8f2a.el8.x8 2.1 MB/s | 846 kB 00:00 2023-03-25T00:55:02.287 INFO:teuthology.orchestra.run.smithi121.stdout:(3/131): ceph-immutable-object-cache-18.0.0-312 1.0 MB/s | 165 kB 00:00 2023-03-25T00:55:02.371 INFO:teuthology.orchestra.run.smithi115.stdout:(3/131): ceph-immutable-object-cache-18.0.0-312 1.1 MB/s | 165 kB 00:00 2023-03-25T00:55:02.723 INFO:teuthology.orchestra.run.smithi121.stdout:(4/131): ceph-base-18.0.0-3124.gc19e8f2a.el8.x8 5.0 MB/s | 5.4 MB 00:01 2023-03-25T00:55:02.746 INFO:teuthology.orchestra.run.smithi115.stdout:(4/131): ceph-base-18.0.0-3124.gc19e8f2a.el8.x8 5.1 MB/s | 5.4 MB 00:01 2023-03-25T00:55:03.024 INFO:teuthology.orchestra.run.smithi121.stdout:(5/131): ceph-mgr-18.0.0-3124.gc19e8f2a.el8.x86 5.2 MB/s | 1.5 MB 00:00 2023-03-25T00:55:03.099 INFO:teuthology.orchestra.run.smithi121.stdout:(6/131): ceph-mds-18.0.0-3124.gc19e8f2a.el8.x86 2.8 MB/s | 2.3 MB 00:00 2023-03-25T00:55:03.113 INFO:teuthology.orchestra.run.smithi115.stdout:(5/131): ceph-mgr-18.0.0-3124.gc19e8f2a.el8.x86 4.2 MB/s | 1.5 MB 00:00 2023-03-25T00:55:03.222 INFO:teuthology.orchestra.run.smithi115.stdout:(6/131): ceph-mds-18.0.0-3124.gc19e8f2a.el8.x86 2.7 MB/s | 2.3 MB 00:00 2023-03-25T00:55:03.857 INFO:teuthology.orchestra.run.smithi121.stdout:(7/131): ceph-mon-18.0.0-3124.gc19e8f2a.el8.x86 5.5 MB/s | 4.6 MB 00:00 2023-03-25T00:55:04.030 INFO:teuthology.orchestra.run.smithi115.stdout:(7/131): ceph-mon-18.0.0-3124.gc19e8f2a.el8.x86 5.0 MB/s | 4.6 MB 00:00 2023-03-25T00:55:05.153 INFO:teuthology.orchestra.run.smithi121.stdout:(8/131): ceph-common-18.0.0-3124.gc19e8f2a.el8. 6.2 MB/s | 22 MB 00:03 2023-03-25T00:55:05.241 INFO:teuthology.orchestra.run.smithi121.stdout:(9/131): ceph-selinux-18.0.0-3124.gc19e8f2a.el8 276 kB/s | 24 kB 00:00 2023-03-25T00:55:05.414 INFO:teuthology.orchestra.run.smithi115.stdout:(8/131): ceph-common-18.0.0-3124.gc19e8f2a.el8. 5.8 MB/s | 22 MB 00:03 2023-03-25T00:55:05.514 INFO:teuthology.orchestra.run.smithi115.stdout:(9/131): ceph-selinux-18.0.0-3124.gc19e8f2a.el8 243 kB/s | 24 kB 00:00 2023-03-25T00:55:06.041 INFO:teuthology.orchestra.run.smithi121.stdout:(10/131): ceph-radosgw-18.0.0-3124.gc19e8f2a.el 6.0 MB/s | 13 MB 00:02 2023-03-25T00:55:06.133 INFO:teuthology.orchestra.run.smithi121.stdout:(11/131): cephfs-mirror-18.0.0-3124.gc19e8f2a.e 2.8 MB/s | 258 kB 00:00 2023-03-25T00:55:06.208 INFO:teuthology.orchestra.run.smithi121.stdout:(12/131): libcephfs-devel-18.0.0-3124.gc19e8f2a 407 kB/s | 30 kB 00:00 2023-03-25T00:55:06.331 INFO:teuthology.orchestra.run.smithi115.stdout:(10/131): ceph-radosgw-18.0.0-3124.gc19e8f2a.el 5.7 MB/s | 13 MB 00:02 2023-03-25T00:55:06.367 INFO:teuthology.orchestra.run.smithi121.stdout:(13/131): libcephfs2-18.0.0-3124.gc19e8f2a.el8. 4.7 MB/s | 769 kB 00:00 2023-03-25T00:55:06.414 INFO:teuthology.orchestra.run.smithi115.stdout:(11/131): cephfs-mirror-18.0.0-3124.gc19e8f2a.e 3.0 MB/s | 258 kB 00:00 2023-03-25T00:55:06.451 INFO:teuthology.orchestra.run.smithi121.stdout:(14/131): libcephsqlite-18.0.0-3124.gc19e8f2a.e 1.6 MB/s | 137 kB 00:00 2023-03-25T00:55:06.490 INFO:teuthology.orchestra.run.smithi115.stdout:(12/131): libcephfs-devel-18.0.0-3124.gc19e8f2a 406 kB/s | 30 kB 00:00 2023-03-25T00:55:06.526 INFO:teuthology.orchestra.run.smithi121.stdout:(15/131): librados-devel-18.0.0-3124.gc19e8f2a. 1.8 MB/s | 139 kB 00:00 2023-03-25T00:55:06.685 INFO:teuthology.orchestra.run.smithi121.stdout:(16/131): libradosstriper1-18.0.0-3124.gc19e8f2 2.9 MB/s | 474 kB 00:00 2023-03-25T00:55:06.721 INFO:teuthology.orchestra.run.smithi115.stdout:(13/131): libcephfs2-18.0.0-3124.gc19e8f2a.el8. 3.3 MB/s | 769 kB 00:00 2023-03-25T00:55:06.807 INFO:teuthology.orchestra.run.smithi115.stdout:(14/131): libcephsqlite-18.0.0-3124.gc19e8f2a.e 1.6 MB/s | 137 kB 00:00 2023-03-25T00:55:06.890 INFO:teuthology.orchestra.run.smithi115.stdout:(15/131): librados-devel-18.0.0-3124.gc19e8f2a. 1.6 MB/s | 139 kB 00:00 2023-03-25T00:55:07.066 INFO:teuthology.orchestra.run.smithi115.stdout:(16/131): libradosstriper1-18.0.0-3124.gc19e8f2 2.6 MB/s | 474 kB 00:00 2023-03-25T00:55:07.926 INFO:teuthology.orchestra.run.smithi121.stdout:(17/131): librgw2-18.0.0-3124.gc19e8f2a.el8.x86 5.2 MB/s | 6.5 MB 00:01 2023-03-25T00:55:08.026 INFO:teuthology.orchestra.run.smithi121.stdout:(18/131): python3-ceph-argparse-18.0.0-3124.gc1 449 kB/s | 45 kB 00:00 2023-03-25T00:55:08.119 INFO:teuthology.orchestra.run.smithi121.stdout:(19/131): python3-ceph-common-18.0.0-3124.gc19e 1.2 MB/s | 117 kB 00:00 2023-03-25T00:55:08.210 INFO:teuthology.orchestra.run.smithi121.stdout:(20/131): python3-cephfs-18.0.0-3124.gc19e8f2a. 2.0 MB/s | 190 kB 00:00 2023-03-25T00:55:08.311 INFO:teuthology.orchestra.run.smithi121.stdout:(21/131): python3-rados-18.0.0-3124.gc19e8f2a.e 3.6 MB/s | 370 kB 00:00 2023-03-25T00:55:08.386 INFO:teuthology.orchestra.run.smithi121.stdout:(22/131): python3-rbd-18.0.0-3124.gc19e8f2a.el8 4.8 MB/s | 364 kB 00:00 2023-03-25T00:55:08.391 INFO:teuthology.orchestra.run.smithi115.stdout:(17/131): librgw2-18.0.0-3124.gc19e8f2a.el8.x86 4.9 MB/s | 6.5 MB 00:01 2023-03-25T00:55:08.453 INFO:teuthology.orchestra.run.smithi121.stdout:(23/131): python3-rgw-18.0.0-3124.gc19e8f2a.el8 1.6 MB/s | 107 kB 00:00 2023-03-25T00:55:08.466 INFO:teuthology.orchestra.run.smithi115.stdout:(18/131): python3-ceph-argparse-18.0.0-3124.gc1 597 kB/s | 45 kB 00:00 2023-03-25T00:55:08.520 INFO:teuthology.orchestra.run.smithi121.stdout:(24/131): rbd-fuse-18.0.0-3124.gc19e8f2a.el8.x8 1.4 MB/s | 94 kB 00:00 2023-03-25T00:55:08.558 INFO:teuthology.orchestra.run.smithi115.stdout:(19/131): python3-ceph-common-18.0.0-3124.gc19e 1.2 MB/s | 117 kB 00:00 2023-03-25T00:55:08.659 INFO:teuthology.orchestra.run.smithi115.stdout:(20/131): python3-cephfs-18.0.0-3124.gc19e8f2a. 1.9 MB/s | 190 kB 00:00 2023-03-25T00:55:08.767 INFO:teuthology.orchestra.run.smithi115.stdout:(21/131): python3-rados-18.0.0-3124.gc19e8f2a.e 3.3 MB/s | 370 kB 00:00 2023-03-25T00:55:08.859 INFO:teuthology.orchestra.run.smithi115.stdout:(22/131): python3-rbd-18.0.0-3124.gc19e8f2a.el8 3.9 MB/s | 364 kB 00:00 2023-03-25T00:55:09.076 INFO:teuthology.orchestra.run.smithi115.stdout:(23/131): python3-rgw-18.0.0-3124.gc19e8f2a.el8 492 kB/s | 107 kB 00:00 2023-03-25T00:55:09.129 INFO:teuthology.orchestra.run.smithi121.stdout:(25/131): rbd-mirror-18.0.0-3124.gc19e8f2a.el8. 5.7 MB/s | 3.4 MB 00:00 2023-03-25T00:55:09.151 INFO:teuthology.orchestra.run.smithi115.stdout:(24/131): rbd-fuse-18.0.0-3124.gc19e8f2a.el8.x8 1.2 MB/s | 94 kB 00:00 2023-03-25T00:55:09.196 INFO:teuthology.orchestra.run.smithi121.stdout:(26/131): rbd-nbd-18.0.0-3124.gc19e8f2a.el8.x86 2.6 MB/s | 179 kB 00:00 2023-03-25T00:55:09.304 INFO:teuthology.orchestra.run.smithi121.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3124.g 213 kB/s | 23 kB 00:00 2023-03-25T00:55:09.388 INFO:teuthology.orchestra.run.smithi121.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3124.gc19e8f2 1.5 MB/s | 123 kB 00:00 2023-03-25T00:55:09.510 INFO:teuthology.orchestra.run.smithi115.stdout:(25/131): ceph-osd-18.0.0-3124.gc19e8f2a.el8.x8 2.8 MB/s | 18 MB 00:06 2023-03-25T00:55:09.577 INFO:teuthology.orchestra.run.smithi115.stdout:(26/131): rbd-nbd-18.0.0-3124.gc19e8f2a.el8.x86 2.6 MB/s | 179 kB 00:00 2023-03-25T00:55:09.636 INFO:teuthology.orchestra.run.smithi115.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3124.g 397 kB/s | 23 kB 00:00 2023-03-25T00:55:09.753 INFO:teuthology.orchestra.run.smithi115.stdout:(28/131): rbd-mirror-18.0.0-3124.gc19e8f2a.el8. 5.7 MB/s | 3.4 MB 00:00 2023-03-25T00:55:09.778 INFO:teuthology.orchestra.run.smithi115.stdout:(29/131): ceph-mgr-cephadm-18.0.0-3124.gc19e8f2 870 kB/s | 123 kB 00:00 2023-03-25T00:55:09.872 INFO:teuthology.orchestra.run.smithi121.stdout:(29/131): ceph-test-18.0.0-3124.gc19e8f2a.el8.x 8.8 MB/s | 41 MB 00:04 2023-03-25T00:55:09.970 INFO:teuthology.orchestra.run.smithi115.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3124.gc19e8 7.8 MB/s | 1.7 MB 00:00 2023-03-25T00:55:10.037 INFO:teuthology.orchestra.run.smithi115.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3124.gc1 3.5 MB/s | 239 kB 00:00 2023-03-25T00:55:10.064 INFO:teuthology.orchestra.run.smithi121.stdout:(30/131): ceph-osd-18.0.0-3124.gc19e8f2a.el8.x8 2.5 MB/s | 18 MB 00:06 2023-03-25T00:55:10.079 INFO:teuthology.orchestra.run.smithi115.stdout:(32/131): ceph-mgr-rook-18.0.0-3124.gc19e8f2a.e 1.1 MB/s | 47 kB 00:00 2023-03-25T00:55:10.106 INFO:teuthology.orchestra.run.smithi121.stdout:(31/131): ceph-mgr-dashboard-18.0.0-3124.gc19e8 2.4 MB/s | 1.7 MB 00:00 2023-03-25T00:55:10.129 INFO:teuthology.orchestra.run.smithi115.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3124.gc 291 kB/s | 15 kB 00:00 2023-03-25T00:55:10.181 INFO:teuthology.orchestra.run.smithi121.stdout:(32/131): ceph-mgr-rook-18.0.0-3124.gc19e8f2a.e 634 kB/s | 47 kB 00:00 2023-03-25T00:55:10.206 INFO:teuthology.orchestra.run.smithi121.stdout:(33/131): ceph-mgr-modules-core-18.0.0-3124.gc1 1.6 MB/s | 239 kB 00:00 2023-03-25T00:55:10.229 INFO:teuthology.orchestra.run.smithi115.stdout:(34/131): ceph-volume-18.0.0-3124.gc19e8f2a.el8 2.6 MB/s | 263 kB 00:00 2023-03-25T00:55:10.248 INFO:teuthology.orchestra.run.smithi121.stdout:(34/131): ceph-prometheus-alerts-18.0.0-3124.gc 218 kB/s | 15 kB 00:00 2023-03-25T00:55:10.296 INFO:teuthology.orchestra.run.smithi115.stdout:(35/131): cephadm-18.0.0-3124.gc19e8f2a.el8.noa 1.2 MB/s | 83 kB 00:00 2023-03-25T00:55:10.323 INFO:teuthology.orchestra.run.smithi121.stdout:(35/131): ceph-volume-18.0.0-3124.gc19e8f2a.el8 2.2 MB/s | 263 kB 00:00 2023-03-25T00:55:10.349 INFO:teuthology.orchestra.run.smithi121.stdout:(36/131): cephadm-18.0.0-3124.gc19e8f2a.el8.noa 829 kB/s | 83 kB 00:00 2023-03-25T00:55:10.354 INFO:teuthology.orchestra.run.smithi115.stdout:(36/131): cephfs-top-18.0.0-3124.gc19e8f2a.el8. 310 kB/s | 18 kB 00:00 2023-03-25T00:55:10.391 INFO:teuthology.orchestra.run.smithi121.stdout:(37/131): cephfs-top-18.0.0-3124.gc19e8f2a.el8. 270 kB/s | 18 kB 00:00 2023-03-25T00:55:10.524 INFO:teuthology.orchestra.run.smithi121.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 442 kB/s | 59 kB 00:00 2023-03-25T00:55:10.555 INFO:teuthology.orchestra.run.smithi115.stdout:(37/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.5 MB/s | 306 kB 00:00 2023-03-25T00:55:10.557 INFO:teuthology.orchestra.run.smithi121.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 1.4 MB/s | 306 kB 00:00 2023-03-25T00:55:10.597 INFO:teuthology.orchestra.run.smithi115.stdout:(38/131): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-03-25T00:55:10.641 INFO:teuthology.orchestra.run.smithi121.stdout:(40/131): ceph-mgr-diskprediction-local-18.0.0- 9.6 MB/s | 7.4 MB 00:00 2023-03-25T00:55:10.647 INFO:teuthology.orchestra.run.smithi115.stdout:(39/131): libunwind-1.3.1-3.el8.x86_64.rpm 1.5 MB/s | 75 kB 00:00 2023-03-25T00:55:10.667 INFO:teuthology.orchestra.run.smithi121.stdout:(41/131): libunwind-1.3.1-3.el8.x86_64.rpm 528 kB/s | 75 kB 00:00 2023-03-25T00:55:10.692 INFO:teuthology.orchestra.run.smithi121.stdout:(42/131): python3-asyncssh-2.7.0-2.el8.noarch.r 3.5 MB/s | 476 kB 00:00 2023-03-25T00:55:10.734 INFO:teuthology.orchestra.run.smithi121.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 794 kB/s | 33 kB 00:00 2023-03-25T00:55:10.756 INFO:teuthology.orchestra.run.smithi115.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 4.3 MB/s | 476 kB 00:00 2023-03-25T00:55:10.759 INFO:teuthology.orchestra.run.smithi121.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 2.0 MB/s | 185 kB 00:00 2023-03-25T00:55:10.776 INFO:teuthology.orchestra.run.smithi121.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 390 kB/s | 16 kB 00:00 2023-03-25T00:55:10.793 INFO:teuthology.orchestra.run.smithi121.stdout:(46/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 288 kB/s | 44 kB 00:00 2023-03-25T00:55:10.806 INFO:teuthology.orchestra.run.smithi115.stdout:(41/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 876 kB/s | 44 kB 00:00 2023-03-25T00:55:10.818 INFO:teuthology.orchestra.run.smithi121.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 2.9 MB/s | 173 kB 00:00 2023-03-25T00:55:10.860 INFO:teuthology.orchestra.run.smithi121.stdout:(48/131): python3-cherrypy-18.4.0-1.el8.noarch. 4.5 MB/s | 384 kB 00:00 2023-03-25T00:55:10.865 INFO:teuthology.orchestra.run.smithi115.stdout:(42/131): python3-beautifulsoup4-4.6.3-2.el8.1. 3.1 MB/s | 185 kB 00:00 2023-03-25T00:55:10.877 INFO:teuthology.orchestra.run.smithi121.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 190 kB/s | 11 kB 00:00 2023-03-25T00:55:10.915 INFO:teuthology.orchestra.run.smithi115.stdout:(43/131): python3-cachetools-3.1.1-4.el8.noarch 661 kB/s | 33 kB 00:00 2023-03-25T00:55:10.929 INFO:teuthology.orchestra.run.smithi121.stdout:(50/131): python3-google-auth-1.1.1-10.el8.noar 596 kB/s | 81 kB 00:00 2023-03-25T00:55:10.955 INFO:teuthology.orchestra.run.smithi121.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 186 kB/s | 18 kB 00:00 2023-03-25T00:55:10.988 INFO:teuthology.orchestra.run.smithi121.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 836 kB/s | 49 kB 00:00 2023-03-25T00:55:11.014 INFO:teuthology.orchestra.run.smithi121.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.0 MB/s | 59 kB 00:00 2023-03-25T00:55:11.039 INFO:teuthology.orchestra.run.smithi121.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.2 MB/s | 60 kB 00:00 2023-03-25T00:55:11.072 INFO:teuthology.orchestra.run.smithi121.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 6.0 MB/s | 1.2 MB 00:00 2023-03-25T00:55:11.098 INFO:teuthology.orchestra.run.smithi121.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 267 kB/s | 16 kB 00:00 2023-03-25T00:55:11.123 INFO:teuthology.orchestra.run.smithi121.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.5 MB/s | 283 kB 00:00 2023-03-25T00:55:11.149 INFO:teuthology.orchestra.run.smithi121.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 446 kB/s | 34 kB 00:00 2023-03-25T00:55:11.174 INFO:teuthology.orchestra.run.smithi121.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2023-03-25T00:55:11.199 INFO:teuthology.orchestra.run.smithi121.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-03-25T00:55:11.224 INFO:teuthology.orchestra.run.smithi121.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 248 kB/s | 19 kB 00:00 2023-03-25T00:55:11.249 INFO:teuthology.orchestra.run.smithi121.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 316 kB/s | 24 kB 00:00 2023-03-25T00:55:11.274 INFO:teuthology.orchestra.run.smithi121.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 387 kB/s | 29 kB 00:00 2023-03-25T00:55:11.300 INFO:teuthology.orchestra.run.smithi121.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 365 kB/s | 27 kB 00:00 2023-03-25T00:55:11.325 INFO:teuthology.orchestra.run.smithi121.stdout:(65/131): python3-waitress-1.2.1-2.el8.1.noarch 3.1 MB/s | 241 kB 00:00 2023-03-25T00:55:11.332 INFO:teuthology.orchestra.run.smithi115.stdout:(44/131): ceph-test-18.0.0-3124.gc19e8f2a.el8.x 7.0 MB/s | 41 MB 00:05 2023-03-25T00:55:11.350 INFO:teuthology.orchestra.run.smithi121.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-03-25T00:55:11.357 INFO:teuthology.orchestra.run.smithi115.stdout:(45/131): python3-certifi-2018.10.15-7.el8.noar 37 kB/s | 16 kB 00:00 2023-03-25T00:55:11.376 INFO:teuthology.orchestra.run.smithi121.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-03-25T00:55:11.393 INFO:teuthology.orchestra.run.smithi121.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 1.2 MB/s | 85 kB 00:00 2023-03-25T00:55:11.426 INFO:teuthology.orchestra.run.smithi121.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 304 kB/s | 23 kB 00:00 2023-03-25T00:55:11.443 INFO:teuthology.orchestra.run.smithi121.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-03-25T00:55:11.458 INFO:teuthology.orchestra.run.smithi115.stdout:(46/131): ceph-mgr-diskprediction-local-18.0.0- 4.4 MB/s | 7.4 MB 00:01 2023-03-25T00:55:11.483 INFO:teuthology.orchestra.run.smithi115.stdout:(47/131): python3-cherrypy-18.4.0-1.el8.noarch. 3.0 MB/s | 384 kB 00:00 2023-03-25T00:55:11.494 INFO:teuthology.orchestra.run.smithi121.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 701 kB/s | 47 kB 00:00 2023-03-25T00:55:11.508 INFO:teuthology.orchestra.run.smithi115.stdout:(48/131): python3-cheroot-8.5.2-1.el8.noarch.rp 983 kB/s | 173 kB 00:00 2023-03-25T00:55:11.518 INFO:teuthology.orchestra.run.smithi121.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 2.8 MB/s | 200 kB 00:00 2023-03-25T00:55:11.533 INFO:teuthology.orchestra.run.smithi115.stdout:(49/131): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-03-25T00:55:11.544 INFO:teuthology.orchestra.run.smithi121.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 853 kB/s | 43 kB 00:00 2023-03-25T00:55:11.559 INFO:teuthology.orchestra.run.smithi115.stdout:(50/131): python3-jaraco-functools-2.0-4.el8.no 350 kB/s | 18 kB 00:00 2023-03-25T00:55:11.569 INFO:teuthology.orchestra.run.smithi121.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-03-25T00:55:11.594 INFO:teuthology.orchestra.run.smithi121.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 958 kB/s | 48 kB 00:00 2023-03-25T00:55:11.609 INFO:teuthology.orchestra.run.smithi115.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 977 kB/s | 49 kB 00:00 2023-03-25T00:55:11.620 INFO:teuthology.orchestra.run.smithi121.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-03-25T00:55:11.626 INFO:teuthology.orchestra.run.smithi115.stdout:(52/131): python3-google-auth-1.1.1-10.el8.noar 485 kB/s | 81 kB 00:00 2023-03-25T00:55:11.645 INFO:teuthology.orchestra.run.smithi121.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 852 kB/s | 43 kB 00:00 2023-03-25T00:55:11.659 INFO:teuthology.orchestra.run.smithi115.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.1 MB/s | 59 kB 00:00 2023-03-25T00:55:11.670 INFO:teuthology.orchestra.run.smithi121.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-03-25T00:55:11.685 INFO:teuthology.orchestra.run.smithi115.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.0 MB/s | 60 kB 00:00 2023-03-25T00:55:11.687 INFO:teuthology.orchestra.run.smithi121.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 5.3 MB/s | 229 kB 00:00 2023-03-25T00:55:11.721 INFO:teuthology.orchestra.run.smithi121.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 11 MB/s | 560 kB 00:00 2023-03-25T00:55:11.738 INFO:teuthology.orchestra.run.smithi121.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 372 kB/s | 19 kB 00:00 2023-03-25T00:55:11.752 INFO:teuthology.orchestra.run.smithi115.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 5.3 MB/s | 1.2 MB 00:00 2023-03-25T00:55:11.755 INFO:teuthology.orchestra.run.smithi121.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 4.5 MB/s | 155 kB 00:00 2023-03-25T00:55:11.777 INFO:teuthology.orchestra.run.smithi115.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 170 kB/s | 16 kB 00:00 2023-03-25T00:55:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 11 MB/s | 559 kB 00:00 2023-03-25T00:55:11.802 INFO:teuthology.orchestra.run.smithi115.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.9 MB/s | 283 kB 00:00 2023-03-25T00:55:11.806 INFO:teuthology.orchestra.run.smithi121.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 356 kB/s | 18 kB 00:00 2023-03-25T00:55:11.827 INFO:teuthology.orchestra.run.smithi115.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 446 kB/s | 34 kB 00:00 2023-03-25T00:55:11.853 INFO:teuthology.orchestra.run.smithi115.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.2 MB/s | 61 kB 00:00 2023-03-25T00:55:11.864 INFO:teuthology.orchestra.run.smithi121.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 7.9 MB/s | 605 kB 00:00 2023-03-25T00:55:11.878 INFO:teuthology.orchestra.run.smithi115.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 1.9 MB/s | 196 kB 00:00 2023-03-25T00:55:11.903 INFO:teuthology.orchestra.run.smithi115.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 248 kB/s | 19 kB 00:00 2023-03-25T00:55:11.906 INFO:teuthology.orchestra.run.smithi121.stdout:(86/131): thrift-0.13.0-2.el8.x86_64.rpm 3.3 MB/s | 1.7 MB 00:00 2023-03-25T00:55:11.928 INFO:teuthology.orchestra.run.smithi115.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 317 kB/s | 24 kB 00:00 2023-03-25T00:55:11.931 INFO:teuthology.orchestra.run.smithi121.stdout:(87/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 1.8 MB/s | 238 kB 00:00 2023-03-25T00:55:11.954 INFO:teuthology.orchestra.run.smithi115.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 388 kB/s | 29 kB 00:00 2023-03-25T00:55:11.979 INFO:teuthology.orchestra.run.smithi115.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 364 kB/s | 27 kB 00:00 2023-03-25T00:55:11.982 INFO:teuthology.orchestra.run.smithi121.stdout:(88/131): fuse-2.9.7-15.el8.x86_64.rpm 1.8 MB/s | 83 kB 00:00 2023-03-25T00:55:11.996 INFO:teuthology.orchestra.run.smithi115.stdout:(65/131): python3-waitress-1.2.1-2.el8.1.noarch 3.5 MB/s | 241 kB 00:00 2023-03-25T00:55:12.007 INFO:teuthology.orchestra.run.smithi121.stdout:(89/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 720 kB/s | 72 kB 00:00 2023-03-25T00:55:12.024 INFO:teuthology.orchestra.run.smithi121.stdout:(90/131): elfutils-libelf-devel-0.186-1.el8.x86 1.4 MB/s | 60 kB 00:00 2023-03-25T00:55:12.049 INFO:teuthology.orchestra.run.smithi121.stdout:(91/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 2.3 MB/s | 98 kB 00:00 2023-03-25T00:55:12.051 INFO:teuthology.orchestra.run.smithi115.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 850 kB/s | 61 kB 00:00 2023-03-25T00:55:12.066 INFO:teuthology.orchestra.run.smithi121.stdout:(92/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 1.3 MB/s | 58 kB 00:00 2023-03-25T00:55:12.080 INFO:teuthology.orchestra.run.smithi115.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.0 MB/s | 251 kB 00:00 2023-03-25T00:55:12.083 INFO:teuthology.orchestra.run.smithi121.stdout:(93/131): libcom_err-devel-1.45.6-4.el8.x86_64. 1.1 MB/s | 39 kB 00:00 2023-03-25T00:55:12.105 INFO:teuthology.orchestra.run.smithi115.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 785 kB/s | 85 kB 00:00 2023-03-25T00:55:12.130 INFO:teuthology.orchestra.run.smithi115.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 292 kB/s | 23 kB 00:00 2023-03-25T00:55:12.133 INFO:teuthology.orchestra.run.smithi121.stdout:(94/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 8.6 MB/s | 2.3 MB 00:00 2023-03-25T00:55:12.147 INFO:teuthology.orchestra.run.smithi115.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 70 kB 00:00 2023-03-25T00:55:12.159 INFO:teuthology.orchestra.run.smithi121.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 2.2 MB/s | 171 kB 00:00 2023-03-25T00:55:12.184 INFO:teuthology.orchestra.run.smithi121.stdout:(96/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 5.4 MB/s | 644 kB 00:00 2023-03-25T00:55:12.206 INFO:teuthology.orchestra.run.smithi115.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.0 MB/s | 110 kB 00:00 2023-03-25T00:55:12.209 INFO:teuthology.orchestra.run.smithi121.stdout:(97/131): NetworkManager-initscripts-updown-1.3 1.8 MB/s | 138 kB 00:00 2023-03-25T00:55:12.235 INFO:teuthology.orchestra.run.smithi121.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 2.1 MB/s | 110 kB 00:00 2023-03-25T00:55:12.248 INFO:teuthology.orchestra.run.smithi115.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-03-25T00:55:12.260 INFO:teuthology.orchestra.run.smithi121.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.1 MB/s | 54 kB 00:00 2023-03-25T00:55:12.285 INFO:teuthology.orchestra.run.smithi121.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.0 MB/s | 259 kB 00:00 2023-03-25T00:55:12.290 INFO:teuthology.orchestra.run.smithi115.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.1 MB/s | 259 kB 00:00 2023-03-25T00:55:12.319 INFO:teuthology.orchestra.run.smithi121.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-03-25T00:55:12.323 INFO:teuthology.orchestra.run.smithi115.stdout:(74/131): thrift-0.13.0-2.el8.x86_64.rpm 7.1 MB/s | 1.7 MB 00:00 2023-03-25T00:55:12.335 INFO:teuthology.orchestra.run.smithi121.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 777 kB/s | 39 kB 00:00 2023-03-25T00:55:12.369 INFO:teuthology.orchestra.run.smithi121.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-03-25T00:55:12.380 INFO:teuthology.orchestra.run.smithi115.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 3.5 MB/s | 323 kB 00:00 2023-03-25T00:55:12.403 INFO:teuthology.orchestra.run.smithi121.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.9 MB/s | 538 kB 00:00 2023-03-25T00:55:12.424 INFO:teuthology.orchestra.run.smithi115.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 386 kB/s | 39 kB 00:00 2023-03-25T00:55:12.428 INFO:teuthology.orchestra.run.smithi121.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-03-25T00:55:12.453 INFO:teuthology.orchestra.run.smithi121.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-03-25T00:55:12.458 INFO:teuthology.orchestra.run.smithi115.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 9.3 MB/s | 688 kB 00:00 2023-03-25T00:55:12.492 INFO:teuthology.orchestra.run.smithi115.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.9 MB/s | 538 kB 00:00 2023-03-25T00:55:12.495 INFO:teuthology.orchestra.run.smithi121.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 10 MB/s | 713 kB 00:00 2023-03-25T00:55:12.517 INFO:teuthology.orchestra.run.smithi115.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-03-25T00:55:12.529 INFO:teuthology.orchestra.run.smithi121.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 797 kB/s | 27 kB 00:00 2023-03-25T00:55:12.542 INFO:teuthology.orchestra.run.smithi115.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-03-25T00:55:12.571 INFO:teuthology.orchestra.run.smithi121.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-03-25T00:55:12.575 INFO:teuthology.orchestra.run.smithi115.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 12 MB/s | 713 kB 00:00 2023-03-25T00:55:12.609 INFO:teuthology.orchestra.run.smithi115.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 799 kB/s | 27 kB 00:00 2023-03-25T00:55:12.613 INFO:teuthology.orchestra.run.smithi121.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.0 MB/s | 44 kB 00:00 2023-03-25T00:55:12.659 INFO:teuthology.orchestra.run.smithi115.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-03-25T00:55:12.705 INFO:teuthology.orchestra.run.smithi121.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 19 MB/s | 4.8 MB 00:00 2023-03-25T00:55:12.733 INFO:teuthology.orchestra.run.smithi115.stdout:(84/131): python3-prettytable-0.7.2-14.el8.noar 598 kB/s | 44 kB 00:00 2023-03-25T00:55:12.818 INFO:teuthology.orchestra.run.smithi115.stdout:(85/131): python3-babel-2.5.1-7.el8.noarch.rpm 17 MB/s | 4.8 MB 00:00 2023-03-25T00:55:12.822 INFO:teuthology.orchestra.run.smithi121.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 3.8 MB/s | 457 kB 00:00 2023-03-25T00:55:12.969 INFO:teuthology.orchestra.run.smithi115.stdout:(86/131): python3-scipy-1.0.0-21.module+el8.5.0 16 MB/s | 14 MB 00:00 2023-03-25T00:55:12.972 INFO:teuthology.orchestra.run.smithi121.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 1.5 MB/s | 225 kB 00:00 2023-03-25T00:55:12.994 INFO:teuthology.orchestra.run.smithi115.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 2.6 MB/s | 457 kB 00:00 2023-03-25T00:55:13.014 INFO:teuthology.orchestra.run.smithi121.stdout:(114/131): uuid-1.6.2-43.el8.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2023-03-25T00:55:13.019 INFO:teuthology.orchestra.run.smithi115.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 4.4 MB/s | 225 kB 00:00 2023-03-25T00:55:13.079 INFO:teuthology.orchestra.run.smithi115.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 757 kB/s | 64 kB 00:00 2023-03-25T00:55:13.081 INFO:teuthology.orchestra.run.smithi121.stdout:(115/131): python3-numpy-1.14.3-10.el8.x86_64.r 8.0 MB/s | 3.7 MB 00:00 2023-03-25T00:55:13.103 INFO:teuthology.orchestra.run.smithi115.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.5 MB/s | 126 kB 00:00 2023-03-25T00:55:13.106 INFO:teuthology.orchestra.run.smithi121.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 1.3 MB/s | 126 kB 00:00 2023-03-25T00:55:13.128 INFO:teuthology.orchestra.run.smithi115.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 408 kB/s | 20 kB 00:00 2023-03-25T00:55:13.131 INFO:teuthology.orchestra.run.smithi121.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 398 kB/s | 20 kB 00:00 2023-03-25T00:55:13.148 INFO:teuthology.orchestra.run.smithi121.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.4 MB/s | 103 kB 00:00 2023-03-25T00:55:13.153 INFO:teuthology.orchestra.run.smithi115.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-03-25T00:55:13.182 INFO:teuthology.orchestra.run.smithi121.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 14 MB/s | 709 kB 00:00 2023-03-25T00:55:13.195 INFO:teuthology.orchestra.run.smithi115.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 10 MB/s | 709 kB 00:00 2023-03-25T00:55:13.224 INFO:teuthology.orchestra.run.smithi121.stdout:(120/131): libpq-13.5-1.el8.x86_64.rpm 4.7 MB/s | 198 kB 00:00 2023-03-25T00:55:13.262 INFO:teuthology.orchestra.run.smithi115.stdout:(94/131): python3-numpy-1.14.3-10.el8.x86_64.rp 7.1 MB/s | 3.7 MB 00:00 2023-03-25T00:55:13.287 INFO:teuthology.orchestra.run.smithi115.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 2.1 MB/s | 198 kB 00:00 2023-03-25T00:55:13.329 INFO:teuthology.orchestra.run.smithi115.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-03-25T00:55:13.366 INFO:teuthology.orchestra.run.smithi121.stdout:(121/131): python3-scipy-1.0.0-21.module+el8.5. 11 MB/s | 14 MB 00:01 2023-03-25T00:55:13.371 INFO:teuthology.orchestra.run.smithi115.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-03-25T00:55:13.408 INFO:teuthology.orchestra.run.smithi121.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-03-25T00:55:13.450 INFO:teuthology.orchestra.run.smithi121.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-03-25T00:55:13.463 INFO:teuthology.orchestra.run.smithi115.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 16 MB/s | 4.9 MB 00:00 2023-03-25T00:55:13.530 INFO:teuthology.orchestra.run.smithi115.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-03-25T00:55:13.542 INFO:teuthology.orchestra.run.smithi121.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 12 MB/s | 4.9 MB 00:00 2023-03-25T00:55:13.572 INFO:teuthology.orchestra.run.smithi115.stdout:(100/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-03-25T00:55:13.606 INFO:teuthology.orchestra.run.smithi115.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2023-03-25T00:55:13.625 INFO:teuthology.orchestra.run.smithi121.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2023-03-25T00:55:13.656 INFO:teuthology.orchestra.run.smithi115.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 12 MB/s | 1.5 MB 00:00 2023-03-25T00:55:13.659 INFO:teuthology.orchestra.run.smithi121.stdout:(126/131): postgresql-contrib-10.21-2.module+el 6.8 MB/s | 808 kB 00:00 2023-03-25T00:55:13.681 INFO:teuthology.orchestra.run.smithi115.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.6 MB/s | 200 kB 00:00 2023-03-25T00:55:13.709 INFO:teuthology.orchestra.run.smithi121.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 18 MB/s | 1.5 MB 00:00 2023-03-25T00:55:13.727 INFO:teuthology.orchestra.run.smithi115.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 602 kB/s | 43 kB 00:00 2023-03-25T00:55:13.765 INFO:teuthology.orchestra.run.smithi115.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 2.6 MB/s | 220 kB 00:00 2023-03-25T00:55:13.791 INFO:teuthology.orchestra.run.smithi115.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 760 kB/s | 48 kB 00:00 2023-03-25T00:55:13.816 INFO:teuthology.orchestra.run.smithi115.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-03-25T00:55:13.818 INFO:teuthology.orchestra.run.smithi121.stdout:(128/131): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2023-03-25T00:55:13.833 INFO:teuthology.orchestra.run.smithi115.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 1.0 MB/s | 43 kB 00:00 2023-03-25T00:55:13.858 INFO:teuthology.orchestra.run.smithi115.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 2.5 MB/s | 109 kB 00:00 2023-03-25T00:55:13.868 INFO:teuthology.orchestra.run.smithi121.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-03-25T00:55:13.883 INFO:teuthology.orchestra.run.smithi115.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-03-25T00:55:13.951 INFO:teuthology.orchestra.run.smithi115.stdout:(111/131): postgresql-server-10.21-2.module+el8 8.7 MB/s | 5.1 MB 00:00 2023-03-25T00:55:13.967 INFO:teuthology.orchestra.run.smithi115.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 224 kB/s | 19 kB 00:00 2023-03-25T00:55:13.992 INFO:teuthology.orchestra.run.smithi115.stdout:(113/131): python3-oauthlib-2.1.0-1.el8.noarch. 3.6 MB/s | 155 kB 00:00 2023-03-25T00:55:14.093 INFO:teuthology.orchestra.run.smithi115.stdout:(114/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 2.3 MB/s | 560 kB 00:00 2023-03-25T00:55:14.093 INFO:teuthology.orchestra.run.smithi121.stdout:(130/131): librbd1-18.0.0-3124.gc19e8f2a.el8.x8 9.3 MB/s | 3.6 MB 00:00 2023-03-25T00:55:14.293 INFO:teuthology.orchestra.run.smithi115.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 60 kB/s | 18 kB 00:00 2023-03-25T00:55:14.327 INFO:teuthology.orchestra.run.smithi115.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 1.5 MB/s | 559 kB 00:00 2023-03-25T00:55:14.327 INFO:teuthology.orchestra.run.smithi121.stdout:(131/131): librados2-18.0.0-3124.gc19e8f2a.el8. 5.5 MB/s | 3.7 MB 00:00 2023-03-25T00:55:14.329 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2023-03-25T00:55:14.330 INFO:teuthology.orchestra.run.smithi121.stdout:Total 16 MB/s | 197 MB 00:12 2023-03-25T00:55:14.369 INFO:teuthology.orchestra.run.smithi115.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 2.1 MB/s | 605 kB 00:00 2023-03-25T00:55:14.394 INFO:teuthology.orchestra.run.smithi115.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.3 MB/s | 238 kB 00:00 2023-03-25T00:55:14.459 INFO:teuthology.orchestra.run.smithi115.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.1 MB/s | 72 kB 00:00 2023-03-25T00:55:14.486 INFO:teuthology.orchestra.run.smithi115.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 904 kB/s | 83 kB 00:00 2023-03-25T00:55:14.511 INFO:teuthology.orchestra.run.smithi115.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.1 MB/s | 60 kB 00:00 2023-03-25T00:55:14.536 INFO:teuthology.orchestra.run.smithi115.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-03-25T00:55:14.562 INFO:teuthology.orchestra.run.smithi115.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-03-25T00:55:14.587 INFO:teuthology.orchestra.run.smithi115.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 769 kB/s | 39 kB 00:00 2023-03-25T00:55:14.629 INFO:teuthology.orchestra.run.smithi115.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 9.4 MB/s | 644 kB 00:00 2023-03-25T00:55:14.680 INFO:teuthology.orchestra.run.smithi115.stdout:(126/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.6 MB/s | 2.3 MB 00:00 2023-03-25T00:55:14.705 INFO:teuthology.orchestra.run.smithi115.stdout:(127/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 1.4 MB/s | 171 kB 00:00 2023-03-25T00:55:14.730 INFO:teuthology.orchestra.run.smithi115.stdout:(128/131): NetworkManager-initscripts-updown-1. 1.3 MB/s | 138 kB 00:00 2023-03-25T00:55:14.814 INFO:teuthology.orchestra.run.smithi115.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 9.4 MB/s | 798 kB 00:00 2023-03-25T00:55:15.072 INFO:teuthology.orchestra.run.smithi115.stdout:(130/131): librbd1-18.0.0-3124.gc19e8f2a.el8.x8 9.7 MB/s | 3.6 MB 00:00 2023-03-25T00:55:15.154 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2023-03-25T00:55:15.214 INFO:teuthology.orchestra.run.smithi115.stdout:(131/131): librados2-18.0.0-3124.gc19e8f2a.el8. 6.9 MB/s | 3.7 MB 00:00 2023-03-25T00:55:15.217 INFO:teuthology.orchestra.run.smithi115.stdout:-------------------------------------------------------------------------------- 2023-03-25T00:55:15.217 INFO:teuthology.orchestra.run.smithi115.stdout:Total 15 MB/s | 197 MB 00:13 2023-03-25T00:55:15.273 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2023-03-25T00:55:15.274 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2023-03-25T00:55:16.042 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction check 2023-03-25T00:55:16.162 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction check succeeded. 2023-03-25T00:55:16.162 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction test 2023-03-25T00:55:16.356 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2023-03-25T00:55:16.360 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2023-03-25T00:55:17.259 INFO:teuthology.orchestra.run.smithi115.stdout:Transaction test succeeded. 2023-03-25T00:55:17.263 INFO:teuthology.orchestra.run.smithi115.stdout:Running transaction 2023-03-25T00:55:21.194 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2023-03-25T00:55:21.364 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-25T00:55:21.588 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-25T00:55:21.757 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-25T00:55:21.898 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-25T00:55:22.099 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-25T00:55:22.497 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-25T00:55:22.598 INFO:teuthology.orchestra.run.smithi115.stdout: Preparing : 1/1 2023-03-25T00:55:22.723 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 5/134 2023-03-25T00:55:22.781 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-25T00:55:22.963 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-25T00:55:23.117 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 5/134 2023-03-25T00:55:23.164 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-03-25T00:55:23.290 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-25T00:55:23.338 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-25T00:55:23.417 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-25T00:55:23.471 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-25T00:55:23.704 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-25T00:55:23.848 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-25T00:55:23.949 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-25T00:55:24.111 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-25T00:55:24.120 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-25T00:55:24.309 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-03-25T00:55:24.362 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-25T00:55:24.591 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-25T00:55:24.763 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-25T00:55:24.776 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-25T00:55:24.996 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-25T00:55:25.122 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-25T00:55:25.179 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-25T00:55:25.379 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-25T00:55:25.468 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-25T00:55:25.496 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-25T00:55:25.639 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 14/134 2023-03-25T00:55:25.688 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 9/134 2023-03-25T00:55:25.745 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 14/134 2023-03-25T00:55:25.914 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 15/134 2023-03-25T00:55:25.924 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-25T00:55:26.057 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 15/134 2023-03-25T00:55:26.165 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-25T00:55:26.234 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-03-25T00:55:26.428 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-03-25T00:55:26.574 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-25T00:55:26.600 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-03-25T00:55:26.722 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-25T00:55:26.838 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-25T00:55:26.857 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-25T00:55:26.980 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-25T00:55:27.029 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 14/134 2023-03-25T00:55:27.133 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 14/134 2023-03-25T00:55:27.295 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 15/134 2023-03-25T00:55:27.387 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 15/134 2023-03-25T00:55:27.545 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-25T00:55:27.579 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-25T00:55:27.695 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-25T00:55:27.739 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-25T00:55:27.853 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-25T00:55:28.336 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-25T00:55:28.454 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-25T00:55:28.530 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-03-25T00:55:28.620 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-25T00:55:29.212 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-25T00:55:29.239 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-03-25T00:55:29.386 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-03-25T00:55:29.684 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-03-25T00:55:29.887 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-03-25T00:55:30.106 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-25T00:55:30.115 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-03-25T00:55:30.303 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8. 26/134 2023-03-25T00:55:30.384 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-03-25T00:55:30.416 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el 27/134 2023-03-25T00:55:30.644 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_6 28/134 2023-03-25T00:55:30.846 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-25T00:55:30.902 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-03-25T00:55:31.090 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-03-25T00:55:31.301 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-03-25T00:55:31.547 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-03-25T00:55:31.783 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-25T00:55:31.976 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8. 26/134 2023-03-25T00:55:32.148 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el 27/134 2023-03-25T00:55:32.613 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-25T00:55:33.086 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_6 28/134 2023-03-25T00:55:33.434 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-03-25T00:55:33.639 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-03-25T00:55:33.705 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-03-25T00:55:33.952 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-03-25T00:55:33.998 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-03-25T00:55:34.223 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-03-25T00:55:34.235 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-25T00:55:34.436 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-03-25T00:55:34.641 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-25T00:55:35.108 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-25T00:55:35.309 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 36/134 2023-03-25T00:55:35.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_6 37/134 2023-03-25T00:55:35.594 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-03-25T00:55:35.755 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-03-25T00:55:35.937 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-03-25T00:55:36.001 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-25T00:55:36.047 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-03-25T00:55:36.253 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-03-25T00:55:36.449 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-03-25T00:55:36.579 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-25T00:55:36.646 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-03-25T00:55:36.658 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-25T00:55:36.847 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-25T00:55:36.866 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-03-25T00:55:37.040 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-25T00:55:37.058 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-03-25T00:55:37.227 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-03-25T00:55:37.248 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 36/134 2023-03-25T00:55:37.414 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_6 37/134 2023-03-25T00:55:37.430 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-03-25T00:55:37.589 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-03-25T00:55:37.610 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-03-25T00:55:37.741 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-03-25T00:55:37.802 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-03-25T00:55:37.962 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-03-25T00:55:38.020 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-03-25T00:55:38.151 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-03-25T00:55:38.163 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-03-25T00:55:38.401 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-03-25T00:55:38.483 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-03-25T00:55:38.641 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-03-25T00:55:38.729 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-03-25T00:55:38.989 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-03-25T00:55:39.011 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-03-25T00:55:39.224 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-03-25T00:55:39.337 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-03-25T00:55:39.449 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-03-25T00:55:39.569 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-03-25T00:55:39.778 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-03-25T00:55:39.779 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-03-25T00:55:40.011 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-03-25T00:55:40.055 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-03-25T00:55:40.205 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-03-25T00:55:40.383 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-03-25T00:55:40.398 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-03-25T00:55:40.568 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-03-25T00:55:40.658 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-03-25T00:55:40.731 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-03-25T00:55:40.860 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-03-25T00:55:41.076 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-03-25T00:55:41.130 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-03-25T00:55:41.331 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-03-25T00:55:41.342 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-03-25T00:55:41.503 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-03-25T00:55:41.582 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-03-25T00:55:41.833 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-03-25T00:55:41.837 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-03-25T00:55:41.993 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-25T00:55:42.114 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-03-25T00:55:42.163 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-25T00:55:42.371 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-03-25T00:55:42.423 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-25T00:55:42.640 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-03-25T00:55:42.642 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-25T00:55:42.807 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-03-25T00:55:42.815 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-25T00:55:42.971 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-25T00:55:43.002 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-03-25T00:55:43.152 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-25T00:55:43.175 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-03-25T00:55:43.257 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-03-25T00:55:43.364 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-25T00:55:43.502 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-03-25T00:55:43.556 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-25T00:55:43.730 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-25T00:55:43.736 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-03-25T00:55:43.875 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-25T00:55:43.950 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-25T00:55:44.027 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-25T00:55:44.090 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-25T00:55:44.273 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-03-25T00:55:44.297 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-25T00:55:44.460 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-03-25T00:55:44.500 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-25T00:55:44.684 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-25T00:55:44.783 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-25T00:55:44.922 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-25T00:55:45.018 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-25T00:55:45.192 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-25T00:55:45.194 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-25T00:55:45.415 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-25T00:55:45.419 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-25T00:55:45.579 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-25T00:55:45.727 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-25T00:55:45.960 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-25T00:55:46.154 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-25T00:55:46.378 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-25T00:55:46.566 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-25T00:55:46.585 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-25T00:55:46.730 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-25T00:55:46.805 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-25T00:55:47.004 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-25T00:55:47.034 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-25T00:55:47.220 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-25T00:55:47.335 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-25T00:55:47.447 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-25T00:55:47.625 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-25T00:55:47.729 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-25T00:55:47.823 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-25T00:55:47.973 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-25T00:55:48.141 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-25T00:55:48.401 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-25T00:55:48.583 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-25T00:55:48.751 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 92/134 2023-03-25T00:55:48.805 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 92/134 2023-03-25T00:55:48.968 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 93/134 2023-03-25T00:55:49.537 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-25T00:55:49.827 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-25T00:55:50.107 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-25T00:55:50.377 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-25T00:55:50.672 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-25T00:55:50.857 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-25T00:55:51.669 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-25T00:55:51.856 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 92/134 2023-03-25T00:55:51.937 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 92/134 2023-03-25T00:55:52.156 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 93/134 2023-03-25T00:55:52.537 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-25T00:55:52.758 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-25T00:55:53.345 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-25T00:55:53.639 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 95/134 2023-03-25T00:55:53.715 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 95/134 2023-03-25T00:55:53.866 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 96/134 2023-03-25T00:55:55.827 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-25T00:55:56.081 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-25T00:55:56.667 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-25T00:55:56.981 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 95/134 2023-03-25T00:55:57.200 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 95/134 2023-03-25T00:55:58.025 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 96/134 2023-03-25T00:56:12.492 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 96/134 2023-03-25T00:56:12.492 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2023-03-25T00:56:12.492 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2023-03-25T00:56:12.492 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-03-25T00:56:12.493 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /run 2023-03-25T00:56:12.493 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2023-03-25T00:56:12.493 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2023-03-25T00:56:12.493 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2023-03-25T00:56:12.493 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2023-03-25T00:56:12.493 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2023-03-25T00:56:12.493 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:12.829 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 97/134 2023-03-25T00:56:14.388 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 97/134 2023-03-25T00:56:14.593 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 98/134 2023-03-25T00:56:14.819 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 98/134 2023-03-25T00:56:14.979 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 99/134 2023-03-25T00:56:15.428 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 99/134 2023-03-25T00:56:15.620 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 100/134 2023-03-25T00:56:15.637 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 100/134 2023-03-25T00:56:16.772 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 96/134 2023-03-25T00:56:16.773 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /sys 2023-03-25T00:56:16.773 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /proc 2023-03-25T00:56:16.773 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2023-03-25T00:56:16.773 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /run 2023-03-25T00:56:16.773 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /mnt 2023-03-25T00:56:16.773 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /var/tmp 2023-03-25T00:56:16.774 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /home 2023-03-25T00:56:16.774 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /tmp 2023-03-25T00:56:16.774 INFO:teuthology.orchestra.run.smithi115.stdout:skipping the directory /dev 2023-03-25T00:56:16.774 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:17.126 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 97/134 2023-03-25T00:56:17.272 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 101/134 2023-03-25T00:56:17.503 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 101/134 2023-03-25T00:56:17.618 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.e 102/134 2023-03-25T00:56:17.816 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a. 103/134 2023-03-25T00:56:18.011 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 104/134 2023-03-25T00:56:18.223 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 104/134 2023-03-25T00:56:18.525 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 105/134 2023-03-25T00:56:18.681 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 97/134 2023-03-25T00:56:18.917 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 98/134 2023-03-25T00:56:19.148 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 98/134 2023-03-25T00:56:19.309 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 99/134 2023-03-25T00:56:19.483 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 105/134 2023-03-25T00:56:19.627 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 106/134 2023-03-25T00:56:19.717 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 106/134 2023-03-25T00:56:19.757 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 99/134 2023-03-25T00:56:19.942 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 100/134 2023-03-25T00:56:19.959 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 100/134 2023-03-25T00:56:20.060 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el 107/134 2023-03-25T00:56:20.267 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 108/134 2023-03-25T00:56:20.302 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 108/134 2023-03-25T00:56:20.469 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 109/134 2023-03-25T00:56:20.480 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 109/134 2023-03-25T00:56:21.597 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 101/134 2023-03-25T00:56:21.740 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 110/134 2023-03-25T00:56:21.819 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 101/134 2023-03-25T00:56:21.939 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.e 102/134 2023-03-25T00:56:21.967 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 111/134 2023-03-25T00:56:22.002 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 111/134 2023-03-25T00:56:22.173 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a. 103/134 2023-03-25T00:56:22.186 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 112/134 2023-03-25T00:56:22.229 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 112/134 2023-03-25T00:56:22.357 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 104/134 2023-03-25T00:56:22.402 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 113/134 2023-03-25T00:56:22.571 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 104/134 2023-03-25T00:56:22.747 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 113/134 2023-03-25T00:56:22.883 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 105/134 2023-03-25T00:56:22.902 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 114/134 2023-03-25T00:56:23.840 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 105/134 2023-03-25T00:56:24.007 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 106/134 2023-03-25T00:56:24.099 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 106/134 2023-03-25T00:56:24.452 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el 107/134 2023-03-25T00:56:24.658 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 108/134 2023-03-25T00:56:24.693 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 108/134 2023-03-25T00:56:24.907 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 109/134 2023-03-25T00:56:24.918 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 109/134 2023-03-25T00:56:26.174 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 110/134 2023-03-25T00:56:26.458 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 111/134 2023-03-25T00:56:26.495 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 111/134 2023-03-25T00:56:26.682 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 112/134 2023-03-25T00:56:26.724 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 112/134 2023-03-25T00:56:26.801 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 114/134 2023-03-25T00:56:26.932 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 113/134 2023-03-25T00:56:27.276 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 113/134 2023-03-25T00:56:27.465 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 114/134 2023-03-25T00:56:27.732 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 115/134 2023-03-25T00:56:28.082 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-03-25T00:56:28.274 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 117/134 2023-03-25T00:56:28.500 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-03-25T00:56:28.667 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-03-25T00:56:28.856 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-03-25T00:56:29.105 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-03-25T00:56:29.288 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_ 121/134 2023-03-25T00:56:29.452 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-25T00:56:31.371 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 114/134 2023-03-25T00:56:31.566 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-25T00:56:31.686 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 115/134 2023-03-25T00:56:31.858 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-25T00:56:31.864 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-03-25T00:56:31.876 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-25T00:56:32.058 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-03-25T00:56:32.125 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 124/134 2023-03-25T00:56:32.324 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 125/134 2023-03-25T00:56:32.660 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 126/134 2023-03-25T00:56:32.957 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-03-25T00:56:33.050 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-25T00:56:33.241 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-03-25T00:56:33.295 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-03-25T00:56:33.320 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-03-25T00:56:33.499 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-03-25T00:56:33.510 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 119/134 2023-03-25T00:56:33.512 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-03-25T00:56:33.658 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-25T00:56:33.666 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-25T00:56:33.718 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-25T00:56:33.843 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-03-25T00:56:33.844 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-25T00:56:33.892 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_ 121/134 2023-03-25T00:56:34.044 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-25T00:56:34.044 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-25T00:56:34.081 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-25T00:56:34.281 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-25T00:56:34.473 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-25T00:56:34.761 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-25T00:56:34.761 INFO:teuthology.orchestra.run.smithi121.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-25T00:56:34.761 INFO:teuthology.orchestra.run.smithi121.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-25T00:56:34.761 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:34.977 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 1/134 2023-03-25T00:56:34.977 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2/134 2023-03-25T00:56:34.978 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 3/134 2023-03-25T00:56:34.978 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 4/134 2023-03-25T00:56:34.978 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 5/134 2023-03-25T00:56:34.978 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-25T00:56:34.978 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-25T00:56:34.978 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-25T00:56:34.978 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 9/134 2023-03-25T00:56:34.979 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 10/134 2023-03-25T00:56:34.979 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 11/134 2023-03-25T00:56:34.979 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 12/134 2023-03-25T00:56:34.979 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 13/134 2023-03-25T00:56:34.979 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_ 14/134 2023-03-25T00:56:34.979 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 15/134 2023-03-25T00:56:34.979 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 16/134 2023-03-25T00:56:34.979 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_6 17/134 2023-03-25T00:56:34.980 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 18/134 2023-03-25T00:56:34.980 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 19/134 2023-03-25T00:56:34.980 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el 20/134 2023-03-25T00:56:34.980 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8. 21/134 2023-03-25T00:56:34.980 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_6 22/134 2023-03-25T00:56:34.980 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 23/134 2023-03-25T00:56:34.980 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 24/134 2023-03-25T00:56:34.980 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 25/134 2023-03-25T00:56:34.981 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 26/134 2023-03-25T00:56:34.987 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 27/134 2023-03-25T00:56:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 28/134 2023-03-25T00:56:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a. 29/134 2023-03-25T00:56:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 30/134 2023-03-25T00:56:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 31/134 2023-03-25T00:56:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 32/134 2023-03-25T00:56:34.988 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el 33/134 2023-03-25T00:56:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 34/134 2023-03-25T00:56:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.e 35/134 2023-03-25T00:56:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 36/134 2023-03-25T00:56:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 37/134 2023-03-25T00:56:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 38/134 2023-03-25T00:56:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-25T00:56:34.989 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-25T00:56:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-25T00:56:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-25T00:56:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-25T00:56:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-25T00:56:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-25T00:56:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-25T00:56:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-25T00:56:34.990 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-25T00:56:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-25T00:56:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-25T00:56:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-25T00:56:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-25T00:56:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-25T00:56:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-25T00:56:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-25T00:56:34.991 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-25T00:56:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-25T00:56:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-25T00:56:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-25T00:56:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-25T00:56:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-25T00:56:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-25T00:56:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-25T00:56:34.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-25T00:56:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-25T00:56:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-25T00:56:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-25T00:56:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-25T00:56:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-25T00:56:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-25T00:56:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-25T00:56:34.993 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-03-25T00:56:34.994 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-03-25T00:56:34.994 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-03-25T00:56:34.994 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-03-25T00:56:34.994 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-03-25T00:56:34.995 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-03-25T00:56:34.995 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-03-25T00:56:34.995 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-03-25T00:56:34.995 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-03-25T00:56:34.995 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-03-25T00:56:34.995 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-03-25T00:56:34.995 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-03-25T00:56:34.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-03-25T00:56:34.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-03-25T00:56:34.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-03-25T00:56:34.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-03-25T00:56:34.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-03-25T00:56:34.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-03-25T00:56:34.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-03-25T00:56:34.996 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-03-25T00:56:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-03-25T00:56:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-03-25T00:56:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-03-25T00:56:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-03-25T00:56:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-03-25T00:56:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-03-25T00:56:34.997 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-03-25T00:56:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-03-25T00:56:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-03-25T00:56:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-03-25T00:56:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-03-25T00:56:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-03-25T00:56:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-03-25T00:56:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-03-25T00:56:34.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-03-25T00:56:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-03-25T00:56:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-03-25T00:56:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-03-25T00:56:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-03-25T00:56:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-03-25T00:56:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-03-25T00:56:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-03-25T00:56:34.999 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-03-25T00:56:35.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-03-25T00:56:35.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-03-25T00:56:35.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-03-25T00:56:35.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-03-25T00:56:35.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-03-25T00:56:35.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-03-25T00:56:35.000 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-03-25T00:56:35.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-03-25T00:56:35.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-03-25T00:56:35.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-03-25T00:56:35.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-03-25T00:56:35.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-03-25T00:56:35.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-03-25T00:56:35.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-03-25T00:56:35.001 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 129/134 2023-03-25T00:56:35.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-25T00:56:35.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 131/134 2023-03-25T00:56:35.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-25T00:56:35.002 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-25T00:56:36.295 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-25T00:56:36.588 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-25T00:56:36.607 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-25T00:56:36.896 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 124/134 2023-03-25T00:56:37.113 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 125/134 2023-03-25T00:56:37.408 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 126/134 2023-03-25T00:56:37.644 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-25T00:56:37.644 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:47 ago on Sat 25 Mar 2023 12:54:50 AM UTC. 2023-03-25T00:56:37.756 INFO:teuthology.orchestra.run.smithi115.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-25T00:56:37.888 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-25T00:56:37.897 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-25T00:56:38.111 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-03-25T00:56:38.312 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-03-25T00:56:38.394 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-03-25T00:56:38.623 INFO:teuthology.orchestra.run.smithi115.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-03-25T00:56:38.636 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-03-25T00:56:38.636 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-25T00:56:38.832 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-25T00:56:38.832 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-25T00:56:39.102 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-25T00:56:39.319 INFO:teuthology.orchestra.run.smithi115.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-25T00:56:39.605 INFO:teuthology.orchestra.run.smithi115.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-25T00:56:39.606 INFO:teuthology.orchestra.run.smithi115.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-25T00:56:39.606 INFO:teuthology.orchestra.run.smithi115.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-25T00:56:39.606 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:39.818 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 1/134 2023-03-25T00:56:39.818 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2/134 2023-03-25T00:56:39.819 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 3/134 2023-03-25T00:56:39.819 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 4/134 2023-03-25T00:56:39.819 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 5/134 2023-03-25T00:56:39.819 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-25T00:56:39.819 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-25T00:56:39.819 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-25T00:56:39.819 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 9/134 2023-03-25T00:56:39.820 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 10/134 2023-03-25T00:56:39.820 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 11/134 2023-03-25T00:56:39.820 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 12/134 2023-03-25T00:56:39.820 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 13/134 2023-03-25T00:56:39.820 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_ 14/134 2023-03-25T00:56:39.820 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 15/134 2023-03-25T00:56:39.820 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 16/134 2023-03-25T00:56:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_6 17/134 2023-03-25T00:56:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 18/134 2023-03-25T00:56:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 19/134 2023-03-25T00:56:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el 20/134 2023-03-25T00:56:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8. 21/134 2023-03-25T00:56:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_6 22/134 2023-03-25T00:56:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 23/134 2023-03-25T00:56:39.821 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 24/134 2023-03-25T00:56:39.828 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 25/134 2023-03-25T00:56:39.829 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 26/134 2023-03-25T00:56:39.829 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 27/134 2023-03-25T00:56:39.829 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 28/134 2023-03-25T00:56:39.829 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a. 29/134 2023-03-25T00:56:39.829 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 30/134 2023-03-25T00:56:39.830 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 31/134 2023-03-25T00:56:39.830 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 32/134 2023-03-25T00:56:39.830 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el 33/134 2023-03-25T00:56:39.830 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 34/134 2023-03-25T00:56:39.830 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.e 35/134 2023-03-25T00:56:39.830 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 36/134 2023-03-25T00:56:39.830 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 37/134 2023-03-25T00:56:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 38/134 2023-03-25T00:56:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-25T00:56:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-25T00:56:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-25T00:56:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-25T00:56:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-25T00:56:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-25T00:56:39.831 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-25T00:56:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-25T00:56:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-25T00:56:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-25T00:56:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-25T00:56:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-25T00:56:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-25T00:56:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-25T00:56:39.832 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-25T00:56:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-25T00:56:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-25T00:56:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-25T00:56:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-25T00:56:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-25T00:56:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-25T00:56:39.833 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-25T00:56:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-25T00:56:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-25T00:56:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-25T00:56:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-25T00:56:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-25T00:56:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-25T00:56:39.834 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-25T00:56:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-25T00:56:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-25T00:56:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-25T00:56:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-25T00:56:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-03-25T00:56:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-03-25T00:56:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-03-25T00:56:39.835 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-03-25T00:56:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-03-25T00:56:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-03-25T00:56:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-03-25T00:56:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-03-25T00:56:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-03-25T00:56:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-03-25T00:56:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-03-25T00:56:39.836 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-03-25T00:56:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-03-25T00:56:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-03-25T00:56:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-03-25T00:56:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-03-25T00:56:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-03-25T00:56:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-03-25T00:56:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-03-25T00:56:39.837 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-03-25T00:56:39.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-03-25T00:56:39.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-03-25T00:56:39.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-03-25T00:56:39.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-03-25T00:56:39.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-03-25T00:56:39.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-03-25T00:56:39.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-03-25T00:56:39.838 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-03-25T00:56:39.839 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-03-25T00:56:39.839 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-03-25T00:56:39.839 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-03-25T00:56:39.839 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-03-25T00:56:39.839 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-03-25T00:56:39.839 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-03-25T00:56:39.839 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-03-25T00:56:39.839 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-03-25T00:56:39.840 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-03-25T00:56:39.840 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-03-25T00:56:39.840 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-03-25T00:56:39.840 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-03-25T00:56:39.840 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-03-25T00:56:39.840 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-03-25T00:56:39.840 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-03-25T00:56:39.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-03-25T00:56:39.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-03-25T00:56:39.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-03-25T00:56:39.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-03-25T00:56:39.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-03-25T00:56:39.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-25T00:56:39.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-03-25T00:56:39.841 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-25T00:56:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-03-25T00:56:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-03-25T00:56:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-03-25T00:56:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-03-25T00:56:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-03-25T00:56:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-25T00:56:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 129/134 2023-03-25T00:56:39.842 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-25T00:56:39.843 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 131/134 2023-03-25T00:56:39.843 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-25T00:56:39.843 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-25T00:56:40.222 INFO:teuthology.orchestra.run.smithi121.stdout:Installed products updated. 2023-03-25T00:56:40.439 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:40.440 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2023-03-25T00:56:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-25T00:56:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.440 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.440 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2023-03-25T00:56:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-25T00:56:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: automake-1.16.1-7.el8.noarch 2023-03-25T00:56:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: bison-3.0.4-10.el8.x86_64 2023-03-25T00:56:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.441 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.442 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.443 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-25T00:56:40.444 INFO:teuthology.orchestra.run.smithi121.stdout: flex-2.6.1-9.el8.x86_64 2023-03-25T00:56:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-25T00:56:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-25T00:56:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: jq-1.6-3.el8.x86_64 2023-03-25T00:56:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-25T00:56:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-25T00:56:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-25T00:56:40.445 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-25T00:56:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-25T00:56:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-25T00:56:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: libpq-13.5-1.el8.x86_64 2023-03-25T00:56:40.446 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-25T00:56:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-25T00:56:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-25T00:56:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-25T00:56:40.447 INFO:teuthology.orchestra.run.smithi121.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-25T00:56:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-25T00:56:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-25T00:56:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-25T00:56:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-25T00:56:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-25T00:56:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-25T00:56:40.448 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-25T00:56:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-25T00:56:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-25T00:56:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-25T00:56:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-25T00:56:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-25T00:56:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-25T00:56:40.449 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-25T00:56:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-25T00:56:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-25T00:56:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-25T00:56:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-25T00:56:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-25T00:56:40.450 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-25T00:56:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-25T00:56:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-25T00:56:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-25T00:56:40.451 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-25T00:56:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-25T00:56:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-25T00:56:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-25T00:56:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-25T00:56:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-25T00:56:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-25T00:56:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-25T00:56:40.452 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-25T00:56:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-25T00:56:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-25T00:56:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-25T00:56:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-25T00:56:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-25T00:56:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-25T00:56:40.453 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-25T00:56:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-25T00:56:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-25T00:56:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-25T00:56:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-2.6-1.el8.noarch 2023-03-25T00:56:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-25T00:56:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-25T00:56:40.454 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-25T00:56:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-25T00:56:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-25T00:56:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-25T00:56:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-25T00:56:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-25T00:56:40.455 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-25T00:56:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-25T00:56:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-25T00:56:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-25T00:56:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-25T00:56:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-25T00:56:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-25T00:56:40.456 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-25T00:56:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-25T00:56:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-25T00:56:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-25T00:56:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-25T00:56:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-25T00:56:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.457 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-25T00:56:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-25T00:56:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-25T00:56:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-25T00:56:40.458 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-25T00:56:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-25T00:56:40.459 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:40.459 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2023-03-25T00:56:40.656 DEBUG:teuthology.parallel:result is None 2023-03-25T00:56:42.577 INFO:teuthology.orchestra.run.smithi115.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-25T00:56:42.577 INFO:teuthology.orchestra.run.smithi115.stdout:Last metadata expiration check: 0:01:52 ago on Sat 25 Mar 2023 12:54:50 AM UTC. 2023-03-25T00:56:45.287 INFO:teuthology.orchestra.run.smithi115.stdout:Installed products updated. 2023-03-25T00:56:45.519 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:45.520 INFO:teuthology.orchestra.run.smithi115.stdout:Upgraded: 2023-03-25T00:56:45.520 INFO:teuthology.orchestra.run.smithi115.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-25T00:56:45.520 INFO:teuthology.orchestra.run.smithi115.stdout: librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.521 INFO:teuthology.orchestra.run.smithi115.stdout: librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.521 INFO:teuthology.orchestra.run.smithi115.stdout:Installed: 2023-03-25T00:56:45.521 INFO:teuthology.orchestra.run.smithi115.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-25T00:56:45.521 INFO:teuthology.orchestra.run.smithi115.stdout: automake-1.16.1-7.el8.noarch 2023-03-25T00:56:45.522 INFO:teuthology.orchestra.run.smithi115.stdout: bison-3.0.4-10.el8.x86_64 2023-03-25T00:56:45.522 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.522 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.523 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.523 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.523 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.523 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-immutable-object-cache-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.524 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.524 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.524 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.524 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.524 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.525 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.525 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.525 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.525 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.526 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.526 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.526 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.526 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.526 INFO:teuthology.orchestra.run.smithi115.stdout: ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.526 INFO:teuthology.orchestra.run.smithi115.stdout: cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.527 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.527 INFO:teuthology.orchestra.run.smithi115.stdout: cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-25T00:56:45.527 INFO:teuthology.orchestra.run.smithi115.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-25T00:56:45.527 INFO:teuthology.orchestra.run.smithi115.stdout: flex-2.6.1-9.el8.x86_64 2023-03-25T00:56:45.527 INFO:teuthology.orchestra.run.smithi115.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-25T00:56:45.527 INFO:teuthology.orchestra.run.smithi115.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-25T00:56:45.527 INFO:teuthology.orchestra.run.smithi115.stdout: jq-1.6-3.el8.x86_64 2023-03-25T00:56:45.528 INFO:teuthology.orchestra.run.smithi115.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-25T00:56:45.528 INFO:teuthology.orchestra.run.smithi115.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-25T00:56:45.528 INFO:teuthology.orchestra.run.smithi115.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-25T00:56:45.528 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.528 INFO:teuthology.orchestra.run.smithi115.stdout: libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.528 INFO:teuthology.orchestra.run.smithi115.stdout: libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.528 INFO:teuthology.orchestra.run.smithi115.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-25T00:56:45.529 INFO:teuthology.orchestra.run.smithi115.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-25T00:56:45.529 INFO:teuthology.orchestra.run.smithi115.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-25T00:56:45.529 INFO:teuthology.orchestra.run.smithi115.stdout: libpq-13.5-1.el8.x86_64 2023-03-25T00:56:45.529 INFO:teuthology.orchestra.run.smithi115.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-25T00:56:45.529 INFO:teuthology.orchestra.run.smithi115.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-25T00:56:45.529 INFO:teuthology.orchestra.run.smithi115.stdout: librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.529 INFO:teuthology.orchestra.run.smithi115.stdout: libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-25T00:56:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-25T00:56:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-25T00:56:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-25T00:56:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-25T00:56:45.530 INFO:teuthology.orchestra.run.smithi115.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-25T00:56:45.531 INFO:teuthology.orchestra.run.smithi115.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-25T00:56:45.531 INFO:teuthology.orchestra.run.smithi115.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-25T00:56:45.531 INFO:teuthology.orchestra.run.smithi115.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-25T00:56:45.531 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-25T00:56:45.531 INFO:teuthology.orchestra.run.smithi115.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-25T00:56:45.531 INFO:teuthology.orchestra.run.smithi115.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-25T00:56:45.531 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-25T00:56:45.532 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-25T00:56:45.532 INFO:teuthology.orchestra.run.smithi115.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-25T00:56:45.532 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-25T00:56:45.532 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-25T00:56:45.532 INFO:teuthology.orchestra.run.smithi115.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-25T00:56:45.533 INFO:teuthology.orchestra.run.smithi115.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-25T00:56:45.533 INFO:teuthology.orchestra.run.smithi115.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-25T00:56:45.533 INFO:teuthology.orchestra.run.smithi115.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-25T00:56:45.533 INFO:teuthology.orchestra.run.smithi115.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-25T00:56:45.533 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-25T00:56:45.533 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.534 INFO:teuthology.orchestra.run.smithi115.stdout: python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.534 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.534 INFO:teuthology.orchestra.run.smithi115.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-25T00:56:45.535 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-25T00:56:45.535 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-25T00:56:45.535 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-25T00:56:45.535 INFO:teuthology.orchestra.run.smithi115.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-25T00:56:45.535 INFO:teuthology.orchestra.run.smithi115.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-25T00:56:45.535 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-25T00:56:45.535 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-25T00:56:45.536 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-25T00:56:45.536 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-25T00:56:45.536 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-25T00:56:45.536 INFO:teuthology.orchestra.run.smithi115.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-25T00:56:45.536 INFO:teuthology.orchestra.run.smithi115.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-25T00:56:45.536 INFO:teuthology.orchestra.run.smithi115.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-25T00:56:45.536 INFO:teuthology.orchestra.run.smithi115.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-25T00:56:45.537 INFO:teuthology.orchestra.run.smithi115.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-25T00:56:45.537 INFO:teuthology.orchestra.run.smithi115.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-25T00:56:45.537 INFO:teuthology.orchestra.run.smithi115.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-25T00:56:45.537 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-25T00:56:45.537 INFO:teuthology.orchestra.run.smithi115.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-25T00:56:45.537 INFO:teuthology.orchestra.run.smithi115.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-25T00:56:45.538 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-25T00:56:45.538 INFO:teuthology.orchestra.run.smithi115.stdout: python3-portend-2.6-1.el8.noarch 2023-03-25T00:56:45.538 INFO:teuthology.orchestra.run.smithi115.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-25T00:56:45.538 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-25T00:56:45.538 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-25T00:56:45.538 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-25T00:56:45.538 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-25T00:56:45.539 INFO:teuthology.orchestra.run.smithi115.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-25T00:56:45.539 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.539 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.539 INFO:teuthology.orchestra.run.smithi115.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-25T00:56:45.539 INFO:teuthology.orchestra.run.smithi115.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-25T00:56:45.539 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.539 INFO:teuthology.orchestra.run.smithi115.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-25T00:56:45.540 INFO:teuthology.orchestra.run.smithi115.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-25T00:56:45.540 INFO:teuthology.orchestra.run.smithi115.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-25T00:56:45.540 INFO:teuthology.orchestra.run.smithi115.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-25T00:56:45.540 INFO:teuthology.orchestra.run.smithi115.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-25T00:56:45.540 INFO:teuthology.orchestra.run.smithi115.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-25T00:56:45.540 INFO:teuthology.orchestra.run.smithi115.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-25T00:56:45.540 INFO:teuthology.orchestra.run.smithi115.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-25T00:56:45.541 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-25T00:56:45.541 INFO:teuthology.orchestra.run.smithi115.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-25T00:56:45.541 INFO:teuthology.orchestra.run.smithi115.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-25T00:56:45.541 INFO:teuthology.orchestra.run.smithi115.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-25T00:56:45.541 INFO:teuthology.orchestra.run.smithi115.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-25T00:56:45.541 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.541 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.542 INFO:teuthology.orchestra.run.smithi115.stdout: rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-25T00:56:45.542 INFO:teuthology.orchestra.run.smithi115.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-25T00:56:45.542 INFO:teuthology.orchestra.run.smithi115.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-25T00:56:45.542 INFO:teuthology.orchestra.run.smithi115.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-25T00:56:45.542 INFO:teuthology.orchestra.run.smithi115.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-25T00:56:45.542 INFO:teuthology.orchestra.run.smithi115.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-25T00:56:45.543 INFO:teuthology.orchestra.run.smithi115.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-25T00:56:45.543 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:45.543 INFO:teuthology.orchestra.run.smithi115.stdout:Complete! 2023-03-25T00:56:45.742 DEBUG:teuthology.parallel:result is None 2023-03-25T00:56:45.742 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-25T00:56:45.890 DEBUG:teuthology.orchestra.run.smithi115:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-25T00:56:45.913 INFO:teuthology.orchestra.run.smithi115.stdout:18.0.0-3124.gc19e8f2a.el8 2023-03-25T00:56:45.914 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3124.gc19e8f2a.el8 2023-03-25T00:56:45.914 INFO:teuthology.task.install:The correct ceph version 18.0.0-3124.gc19e8f2a is installed. 2023-03-25T00:56:45.915 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-25T00:56:46.071 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-25T00:56:46.099 INFO:teuthology.orchestra.run.smithi121.stdout:18.0.0-3124.gc19e8f2a.el8 2023-03-25T00:56:46.099 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3124.gc19e8f2a.el8 2023-03-25T00:56:46.100 INFO:teuthology.task.install:The correct ceph version 18.0.0-3124.gc19e8f2a is installed. 2023-03-25T00:56:46.100 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-03-25T00:56:46.101 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:46.101 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-25T00:56:46.141 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:56:46.141 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-25T00:56:46.187 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-03-25T00:56:46.187 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:46.188 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/daemon-helper 2023-03-25T00:56:46.214 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-25T00:56:46.281 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:56:46.281 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2023-03-25T00:56:46.311 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-25T00:56:46.379 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-03-25T00:56:46.380 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:46.380 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-25T00:56:46.406 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-25T00:56:46.472 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:56:46.473 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-25T00:56:46.501 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-25T00:56:46.569 INFO:teuthology.run_tasks:Running task ceph... 2023-03-25T00:56:46.641 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-03-25T00:56:46.642 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 777 /var/log/ceph 2023-03-25T00:56:46.644 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /var/log/ceph 2023-03-25T00:56:46.672 INFO:tasks.ceph:Disabling ceph logrotate... 2023-03-25T00:56:46.673 DEBUG:teuthology.orchestra.run.smithi115:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-25T00:56:46.710 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-25T00:56:46.739 INFO:tasks.ceph:Creating extra log directories... 2023-03-25T00:56:46.740 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-25T00:56:46.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-25T00:56:46.812 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-03-25T00:56:46.813 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-03-25T00:56:46.813 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-03-25T00:56:46.814 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:46.814 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-25T00:56:46.856 DEBUG:teuthology.orchestra.run.smithi115:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-25T00:56:47.014 DEBUG:teuthology.orchestra.run.smithi115:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-25T00:56:47.040 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:56:47.041 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-25T00:56:47.060 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-25T00:56:47.236 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-25T00:56:47.269 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-03-25T00:56:47.270 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-03-25T00:56:47.270 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219658', 'branch': 'wip-pdonnell-testing-20230323.162417', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7219658', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2023-03-24_23:11:30-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20230323.162417'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20230323.162417', 'sha1': '1e7de2f05230c7419acf5ca7f5fd22964e30ab77'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 55, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20230323.162417', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_1e7de2f05230c7419acf5ca7f5fd22964e30ab77/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph.git', 'suite_sha1': '1e7de2f05230c7419acf5ca7f5fd22964e30ab77', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi115.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs=', 'smithi121.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'parallel-gzip', 'teuthology_sha1': 'fe9b6e52013911547168fef4a8506325b6c76139', 'timestamp': '2023-03-24_23:11:30', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.4538'} 2023-03-25T00:56:47.271 DEBUG:teuthology.orchestra.run.smithi115:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-25T00:56:47.273 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-25T00:56:47.321 DEBUG:teuthology.orchestra.run.smithi115:> sudo install -d -m0777 -- /var/run/ceph 2023-03-25T00:56:47.331 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m0777 -- /var/run/ceph 2023-03-25T00:56:47.389 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:47.390 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/scratch_devs of=/dev/stdout 2023-03-25T00:56:47.413 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-25T00:56:47.413 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_1 2023-03-25T00:56:47.469 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-25T00:56:47.470 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-25T00:56:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 366040 Links: 1 2023-03-25T00:56:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-25T00:56:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-03-25T00:56:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-03-25 00:56:39.526221605 +0000 2023-03-25T00:56:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-03-25 00:54:03.417625818 +0000 2023-03-25T00:56:47.470 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-03-25 00:54:03.417625818 +0000 2023-03-25T00:56:47.471 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-03-25T00:56:47.471 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-25T00:56:47.535 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-03-25T00:56:47.535 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-03-25T00:56:47.535 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000210601 s, 2.4 MB/s 2023-03-25T00:56:47.536 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-25T00:56:47.593 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_2 2023-03-25T00:56:47.650 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-25T00:56:47.650 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-25T00:56:47.651 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 363696 Links: 1 2023-03-25T00:56:47.651 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-25T00:56:47.651 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-03-25T00:56:47.651 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-03-25 00:56:39.526221605 +0000 2023-03-25T00:56:47.651 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-03-25 00:54:03.927615806 +0000 2023-03-25T00:56:47.651 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-03-25 00:54:03.927615806 +0000 2023-03-25T00:56:47.651 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-03-25T00:56:47.652 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-25T00:56:47.716 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-03-25T00:56:47.716 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-03-25T00:56:47.716 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.0003017 s, 1.7 MB/s 2023-03-25T00:56:47.717 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-25T00:56:47.775 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_3 2023-03-25T00:56:47.832 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-25T00:56:47.833 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-25T00:56:47.833 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 361874 Links: 1 2023-03-25T00:56:47.833 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-25T00:56:47.833 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-03-25T00:56:47.833 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-03-25 00:56:39.526221605 +0000 2023-03-25T00:56:47.833 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-03-25 00:54:04.475605048 +0000 2023-03-25T00:56:47.834 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-03-25 00:54:04.475605048 +0000 2023-03-25T00:56:47.834 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-03-25T00:56:47.834 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-25T00:56:47.898 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-03-25T00:56:47.898 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-03-25T00:56:47.899 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.00029743 s, 1.7 MB/s 2023-03-25T00:56:47.900 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-25T00:56:47.957 DEBUG:teuthology.orchestra.run.smithi115:> stat /dev/vg_nvme/lv_4 2023-03-25T00:56:48.013 INFO:teuthology.orchestra.run.smithi115.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-25T00:56:48.014 INFO:teuthology.orchestra.run.smithi115.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-25T00:56:48.014 INFO:teuthology.orchestra.run.smithi115.stdout:Device: 6h/6d Inode: 363214 Links: 1 2023-03-25T00:56:48.014 INFO:teuthology.orchestra.run.smithi115.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-25T00:56:48.014 INFO:teuthology.orchestra.run.smithi115.stdout:Context: system_u:object_r:device_t:s0 2023-03-25T00:56:48.014 INFO:teuthology.orchestra.run.smithi115.stdout:Access: 2023-03-25 00:56:39.526221605 +0000 2023-03-25T00:56:48.014 INFO:teuthology.orchestra.run.smithi115.stdout:Modify: 2023-03-25 00:54:04.998594781 +0000 2023-03-25T00:56:48.015 INFO:teuthology.orchestra.run.smithi115.stdout:Change: 2023-03-25 00:54:04.998594781 +0000 2023-03-25T00:56:48.015 INFO:teuthology.orchestra.run.smithi115.stdout: Birth: - 2023-03-25T00:56:48.015 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-25T00:56:48.078 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records in 2023-03-25T00:56:48.078 INFO:teuthology.orchestra.run.smithi115.stderr:1+0 records out 2023-03-25T00:56:48.078 INFO:teuthology.orchestra.run.smithi115.stderr:512 bytes copied, 0.000182775 s, 2.8 MB/s 2023-03-25T00:56:48.079 DEBUG:teuthology.orchestra.run.smithi115:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-25T00:56:48.138 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-25T00:56:48.138 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:56:48.138 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/scratch_devs of=/dev/stdout 2023-03-25T00:56:48.159 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-25T00:56:48.159 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_1 2023-03-25T00:56:48.220 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-25T00:56:48.220 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-25T00:56:48.221 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 364907 Links: 1 2023-03-25T00:56:48.221 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-25T00:56:48.221 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-03-25T00:56:48.222 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-03-25 00:56:34.681852587 +0000 2023-03-25T00:56:48.222 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-03-25 00:54:03.438528306 +0000 2023-03-25T00:56:48.222 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-03-25 00:54:03.438528306 +0000 2023-03-25T00:56:48.222 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-03-25T00:56:48.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-25T00:56:48.289 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-03-25T00:56:48.289 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-03-25T00:56:48.289 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000266924 s, 1.9 MB/s 2023-03-25T00:56:48.290 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-25T00:56:48.350 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_2 2023-03-25T00:56:48.410 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-25T00:56:48.410 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-25T00:56:48.410 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 364936 Links: 1 2023-03-25T00:56:48.410 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-25T00:56:48.410 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-03-25T00:56:48.411 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-03-25 00:56:34.681852587 +0000 2023-03-25T00:56:48.411 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-03-25 00:54:03.947521628 +0000 2023-03-25T00:56:48.411 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-03-25 00:54:03.947521628 +0000 2023-03-25T00:56:48.411 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-03-25T00:56:48.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-25T00:56:48.478 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-03-25T00:56:48.478 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-03-25T00:56:48.478 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000219273 s, 2.3 MB/s 2023-03-25T00:56:48.479 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-25T00:56:48.538 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_3 2023-03-25T00:56:48.596 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-25T00:56:48.597 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-25T00:56:48.597 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 360261 Links: 1 2023-03-25T00:56:48.597 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-25T00:56:48.597 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-03-25T00:56:48.597 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-03-25 00:56:34.681852587 +0000 2023-03-25T00:56:48.597 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-03-25 00:54:04.449515042 +0000 2023-03-25T00:56:48.598 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-03-25 00:54:04.449515042 +0000 2023-03-25T00:56:48.598 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-03-25T00:56:48.598 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-25T00:56:48.663 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-03-25T00:56:48.664 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-03-25T00:56:48.664 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000232625 s, 2.2 MB/s 2023-03-25T00:56:48.665 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-25T00:56:48.723 DEBUG:teuthology.orchestra.run.smithi121:> stat /dev/vg_nvme/lv_4 2023-03-25T00:56:48.781 INFO:teuthology.orchestra.run.smithi121.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-25T00:56:48.782 INFO:teuthology.orchestra.run.smithi121.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-25T00:56:48.782 INFO:teuthology.orchestra.run.smithi121.stdout:Device: 6h/6d Inode: 359210 Links: 1 2023-03-25T00:56:48.782 INFO:teuthology.orchestra.run.smithi121.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-25T00:56:48.782 INFO:teuthology.orchestra.run.smithi121.stdout:Context: system_u:object_r:device_t:s0 2023-03-25T00:56:48.782 INFO:teuthology.orchestra.run.smithi121.stdout:Access: 2023-03-25 00:56:34.681852587 +0000 2023-03-25T00:56:48.782 INFO:teuthology.orchestra.run.smithi121.stdout:Modify: 2023-03-25 00:54:04.968508233 +0000 2023-03-25T00:56:48.783 INFO:teuthology.orchestra.run.smithi121.stdout:Change: 2023-03-25 00:54:04.968508233 +0000 2023-03-25T00:56:48.783 INFO:teuthology.orchestra.run.smithi121.stdout: Birth: - 2023-03-25T00:56:48.783 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-25T00:56:48.846 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records in 2023-03-25T00:56:48.847 INFO:teuthology.orchestra.run.smithi121.stderr:1+0 records out 2023-03-25T00:56:48.847 INFO:teuthology.orchestra.run.smithi121.stderr:512 bytes copied, 0.000190392 s, 2.7 MB/s 2023-03-25T00:56:48.848 DEBUG:teuthology.orchestra.run.smithi121:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-25T00:56:48.905 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-25T00:56:48.906 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-25T00:56:48.906 INFO:tasks.ceph:Generating config... 2023-03-25T00:56:48.907 INFO:tasks.ceph:[client] client mount timeout = 600 2023-03-25T00:56:48.907 INFO:tasks.ceph:[client] debug client = 20 2023-03-25T00:56:48.908 INFO:tasks.ceph:[client] debug ms = 1 2023-03-25T00:56:48.908 INFO:tasks.ceph:[client] pid file = 2023-03-25T00:56:48.908 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-03-25T00:56:48.908 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-03-25T00:56:48.908 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-03-25T00:56:48.909 INFO:tasks.ceph:[mds] debug mds = 20 2023-03-25T00:56:48.909 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-03-25T00:56:48.909 INFO:tasks.ceph:[mds] debug ms = 1 2023-03-25T00:56:48.909 INFO:tasks.ceph:[mds] mds debug frag = True 2023-03-25T00:56:48.909 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-03-25T00:56:48.910 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-03-25T00:56:48.910 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-03-25T00:56:48.910 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-03-25T00:56:48.910 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-03-25T00:56:48.910 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-03-25T00:56:48.911 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-03-25T00:56:48.911 INFO:tasks.ceph:[mgr] debug ms = 1 2023-03-25T00:56:48.911 INFO:tasks.ceph:[mon] debug mon = 20 2023-03-25T00:56:48.911 INFO:tasks.ceph:[mon] debug ms = 1 2023-03-25T00:56:48.911 INFO:tasks.ceph:[mon] debug paxos = 20 2023-03-25T00:56:48.912 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-03-25T00:56:48.912 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-03-25T00:56:48.912 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-03-25T00:56:48.912 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-03-25T00:56:48.912 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-03-25T00:56:48.912 INFO:tasks.ceph:[osd] debug ms = 1 2023-03-25T00:56:48.913 INFO:tasks.ceph:[osd] debug osd = 20 2023-03-25T00:56:48.913 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-03-25T00:56:48.913 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-03-25T00:56:48.913 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-03-25T00:56:48.913 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-03-25T00:56:48.914 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-03-25T00:56:48.914 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-03-25T00:56:48.914 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-03-25T00:56:48.914 INFO:tasks.ceph:Setting up mon.a... 2023-03-25T00:56:48.915 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-03-25T00:56:48.973 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.keyring 2023-03-25T00:56:48.976 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-03-25T00:56:49.024 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-25T00:56:49.051 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.115'), ('mon.b', '172.21.15.121'), ('mon.c', '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]')] 2023-03-25T00:56:49.051 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.115,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'false', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-03-25T00:56:49.052 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:49.053 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-03-25T00:56:49.108 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.115 --add b 172.21.15.121 --addv c '[v2:172.21.15.121:3301,v1:172.21.15.121:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:56:49.195 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:56:49.195 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: generated fsid 9c1ea6b6-7c28-4560-af27-3b583d106388 2023-03-25T00:56:49.195 INFO:teuthology.orchestra.run.smithi115.stdout:setting min_mon_release = pacific 2023-03-25T00:56:49.196 INFO:teuthology.orchestra.run.smithi115.stdout:epoch 0 2023-03-25T00:56:49.196 INFO:teuthology.orchestra.run.smithi115.stdout:fsid 9c1ea6b6-7c28-4560-af27-3b583d106388 2023-03-25T00:56:49.196 INFO:teuthology.orchestra.run.smithi115.stdout:last_changed 2023-03-25T00:56:49.194443+0000 2023-03-25T00:56:49.196 INFO:teuthology.orchestra.run.smithi115.stdout:created 2023-03-25T00:56:49.194443+0000 2023-03-25T00:56:49.196 INFO:teuthology.orchestra.run.smithi115.stdout:min_mon_release 16 (pacific) 2023-03-25T00:56:49.196 INFO:teuthology.orchestra.run.smithi115.stdout:election_strategy: 1 2023-03-25T00:56:49.196 INFO:teuthology.orchestra.run.smithi115.stdout:0: [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] mon.a 2023-03-25T00:56:49.196 INFO:teuthology.orchestra.run.smithi115.stdout:1: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.b 2023-03-25T00:56:49.197 INFO:teuthology.orchestra.run.smithi115.stdout:2: [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] mon.c 2023-03-25T00:56:49.197 INFO:teuthology.orchestra.run.smithi115.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-03-25T00:56:49.198 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-03-25T00:56:49.214 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9c1ea6b6-7c28-4560-af27-3b583d106388... 2023-03-25T00:56:49.215 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-25T00:56:49.256 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-25T00:56:49.298 INFO:teuthology.orchestra.run.smithi115.stdout:[global] 2023-03-25T00:56:49.298 INFO:teuthology.orchestra.run.smithi115.stdout: chdir = "" 2023-03-25T00:56:49.299 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-25T00:56:49.299 INFO:teuthology.orchestra.run.smithi115.stdout: auth supported = cephx 2023-03-25T00:56:49.299 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.299 INFO:teuthology.orchestra.run.smithi115.stdout: filestore xattr use omap = true 2023-03-25T00:56:49.299 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.299 INFO:teuthology.orchestra.run.smithi115.stdout: mon clock drift allowed = 1.000 2023-03-25T00:56:49.299 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.300 INFO:teuthology.orchestra.run.smithi115.stdout: osd crush chooseleaf type = 0 2023-03-25T00:56:49.300 INFO:teuthology.orchestra.run.smithi115.stdout: auth debug = true 2023-03-25T00:56:49.300 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.300 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on old message = true 2023-03-25T00:56:49.300 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on bug = true 2023-03-25T00:56:49.300 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.300 INFO:teuthology.orchestra.run.smithi115.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-25T00:56:49.300 INFO:teuthology.orchestra.run.smithi115.stdout: mon pg warn max object skew = 0 2023-03-25T00:56:49.301 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.301 INFO:teuthology.orchestra.run.smithi115.stdout: # disable pg_autoscaler by default for new pools 2023-03-25T00:56:49.301 INFO:teuthology.orchestra.run.smithi115.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-25T00:56:49.301 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.301 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default size = 2 2023-03-25T00:56:49.301 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.301 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow primary affinity = true 2023-03-25T00:56:49.302 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd allow pg remap = true 2023-03-25T00:56:49.302 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on legacy crush tunables = false 2023-03-25T00:56:49.302 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on crush straw calc version zero = false 2023-03-25T00:56:49.302 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on no sortbitwise = false 2023-03-25T00:56:49.302 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on osd down out interval zero = false 2023-03-25T00:56:49.302 INFO:teuthology.orchestra.run.smithi115.stdout: mon warn on too few osds = false 2023-03-25T00:56:49.302 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-25T00:56:49.302 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-25T00:56:49.303 INFO:teuthology.orchestra.run.smithi115.stdout: mon_allow_pool_size_one = true 2023-03-25T00:56:49.303 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.303 INFO:teuthology.orchestra.run.smithi115.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-25T00:56:49.303 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.303 INFO:teuthology.orchestra.run.smithi115.stdout: osd default data pool replay window = 5 2023-03-25T00:56:49.303 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.303 INFO:teuthology.orchestra.run.smithi115.stdout: mon allow pool delete = true 2023-03-25T00:56:49.303 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.304 INFO:teuthology.orchestra.run.smithi115.stdout: mon cluster log file level = debug 2023-03-25T00:56:49.304 INFO:teuthology.orchestra.run.smithi115.stdout: debug asserts on shutdown = true 2023-03-25T00:56:49.304 INFO:teuthology.orchestra.run.smithi115.stdout: mon health detail to clog = false 2023-03-25T00:56:49.304 INFO:teuthology.orchestra.run.smithi115.stdout: mon host = "172.21.15.115,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2023-03-25T00:56:49.304 INFO:teuthology.orchestra.run.smithi115.stdout: ms die on skipped message = False 2023-03-25T00:56:49.304 INFO:teuthology.orchestra.run.smithi115.stdout: fsid = 9c1ea6b6-7c28-4560-af27-3b583d106388 2023-03-25T00:56:49.304 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.304 INFO:teuthology.orchestra.run.smithi115.stdout:[osd] 2023-03-25T00:56:49.305 INFO:teuthology.orchestra.run.smithi115.stdout: osd journal size = 100 2023-03-25T00:56:49.305 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.305 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub load threshold = 5.0 2023-03-25T00:56:49.305 INFO:teuthology.orchestra.run.smithi115.stdout: osd scrub max interval = 600 2023-03-25T00:56:49.305 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.305 INFO:teuthology.orchestra.run.smithi115.stdout: osd recover clone overlap = true 2023-03-25T00:56:49.305 INFO:teuthology.orchestra.run.smithi115.stdout: osd recovery max chunk = 1048576 2023-03-25T00:56:49.305 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.306 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug shutdown = true 2023-03-25T00:56:49.306 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug op order = false 2023-03-25T00:56:49.306 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug verify stray on activate = true 2023-03-25T00:56:49.306 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.306 INFO:teuthology.orchestra.run.smithi115.stdout: osd open classes on start = true 2023-03-25T00:56:49.306 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug pg log writeout = true 2023-03-25T00:56:49.306 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.306 INFO:teuthology.orchestra.run.smithi115.stdout: osd deep scrub update digest min age = 30 2023-03-25T00:56:49.307 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.307 INFO:teuthology.orchestra.run.smithi115.stdout: osd map max advance = 10 2023-03-25T00:56:49.307 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.307 INFO:teuthology.orchestra.run.smithi115.stdout: journal zero on create = true 2023-03-25T00:56:49.307 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.307 INFO:teuthology.orchestra.run.smithi115.stdout: filestore ondisk finisher threads = 3 2023-03-25T00:56:49.307 INFO:teuthology.orchestra.run.smithi115.stdout: filestore apply finisher threads = 3 2023-03-25T00:56:49.308 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.308 INFO:teuthology.orchestra.run.smithi115.stdout: bdev debug aio = true 2023-03-25T00:56:49.308 INFO:teuthology.orchestra.run.smithi115.stdout: osd debug misdirected ops = true 2023-03-25T00:56:49.308 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore block size = 96636764160 2023-03-25T00:56:49.308 INFO:teuthology.orchestra.run.smithi115.stdout: bluestore fsck on mount = True 2023-03-25T00:56:49.308 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluefs = 20 2023-03-25T00:56:49.308 INFO:teuthology.orchestra.run.smithi115.stdout: debug bluestore = 20 2023-03-25T00:56:49.308 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-03-25T00:56:49.309 INFO:teuthology.orchestra.run.smithi115.stdout: debug osd = 20 2023-03-25T00:56:49.309 INFO:teuthology.orchestra.run.smithi115.stdout: debug rocksdb = 10 2023-03-25T00:56:49.309 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd backfillfull_ratio = 0.85 2023-03-25T00:56:49.309 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd full ratio = 0.9 2023-03-25T00:56:49.309 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd nearfull ratio = 0.8 2023-03-25T00:56:49.309 INFO:teuthology.orchestra.run.smithi115.stdout: osd failsafe full ratio = 0.95 2023-03-25T00:56:49.309 INFO:teuthology.orchestra.run.smithi115.stdout: osd objectstore = bluestore 2023-03-25T00:56:49.309 INFO:teuthology.orchestra.run.smithi115.stdout: osd op complaint time = 180 2023-03-25T00:56:49.310 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.310 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr] 2023-03-25T00:56:49.310 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-03-25T00:56:49.310 INFO:teuthology.orchestra.run.smithi115.stdout: debug mgr = 20 2023-03-25T00:56:49.310 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2023-03-25T00:56:49.310 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2023-03-25T00:56:49.310 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2023-03-25T00:56:49.310 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2023-03-25T00:56:49.311 INFO:teuthology.orchestra.run.smithi115.stdout: mgr/telemetry/nag = false 2023-03-25T00:56:49.311 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.311 INFO:teuthology.orchestra.run.smithi115.stdout:[mon] 2023-03-25T00:56:49.311 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-03-25T00:56:49.311 INFO:teuthology.orchestra.run.smithi115.stdout: debug mon = 20 2023-03-25T00:56:49.311 INFO:teuthology.orchestra.run.smithi115.stdout: debug paxos = 20 2023-03-25T00:56:49.311 INFO:teuthology.orchestra.run.smithi115.stdout: debug auth = 20 2023-03-25T00:56:49.311 INFO:teuthology.orchestra.run.smithi115.stdout: mon data avail warn = 5 2023-03-25T00:56:49.312 INFO:teuthology.orchestra.run.smithi115.stdout: mon mgr mkfs grace = 240 2023-03-25T00:56:49.312 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min pgs per osd = 4 2023-03-25T00:56:49.312 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd reporter subtree level = osd 2023-03-25T00:56:49.312 INFO:teuthology.orchestra.run.smithi115.stdout: mon osd prime pg temp = true 2023-03-25T00:56:49.312 INFO:teuthology.orchestra.run.smithi115.stdout: mon reweight min bytes per osd = 10 2023-03-25T00:56:49.312 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.312 INFO:teuthology.orchestra.run.smithi115.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-25T00:56:49.312 INFO:teuthology.orchestra.run.smithi115.stdout: auth mon ticket ttl = 660 # 11m 2023-03-25T00:56:49.313 INFO:teuthology.orchestra.run.smithi115.stdout: auth service ticket ttl = 240 # 4m 2023-03-25T00:56:49.313 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.313 INFO:teuthology.orchestra.run.smithi115.stdout: # don't complain about insecure global_id in the test suite 2023-03-25T00:56:49.313 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-25T00:56:49.313 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-25T00:56:49.313 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.313 INFO:teuthology.orchestra.run.smithi115.stdout: # 1m isn't quite enough 2023-03-25T00:56:49.313 INFO:teuthology.orchestra.run.smithi115.stdout: mon_down_mkfs_grace = 2m 2023-03-25T00:56:49.314 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.314 INFO:teuthology.orchestra.run.smithi115.stdout: mon_warn_on_filestore_osds = false 2023-03-25T00:56:49.314 INFO:teuthology.orchestra.run.smithi115.stdout: mon op complaint time = 120 2023-03-25T00:56:49.314 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:56:49.314 INFO:teuthology.orchestra.run.smithi115.stdout:[client] 2023-03-25T00:56:49.314 INFO:teuthology.orchestra.run.smithi115.stdout: rgw cache enabled = true 2023-03-25T00:56:49.314 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable ops log = true 2023-03-25T00:56:49.314 INFO:teuthology.orchestra.run.smithi115.stdout: rgw enable usage log = true 2023-03-25T00:56:49.315 INFO:teuthology.orchestra.run.smithi115.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-25T00:56:49.315 INFO:teuthology.orchestra.run.smithi115.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-25T00:56:49.315 INFO:teuthology.orchestra.run.smithi115.stdout: client mount timeout = 600 2023-03-25T00:56:49.315 INFO:teuthology.orchestra.run.smithi115.stdout: debug client = 20 2023-03-25T00:56:49.315 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-03-25T00:56:49.315 INFO:teuthology.orchestra.run.smithi115.stdout: pid file = "" 2023-03-25T00:56:49.315 INFO:teuthology.orchestra.run.smithi115.stdout: rados mon op timeout = 900 2023-03-25T00:56:49.315 INFO:teuthology.orchestra.run.smithi115.stdout: rados osd op timeout = 900 2023-03-25T00:56:49.316 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.a] 2023-03-25T00:56:49.316 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.b] 2023-03-25T00:56:49.316 INFO:teuthology.orchestra.run.smithi115.stdout:[mon.c] 2023-03-25T00:56:49.316 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.a] 2023-03-25T00:56:49.316 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.c] 2023-03-25T00:56:49.316 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.b] 2023-03-25T00:56:49.316 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.d] 2023-03-25T00:56:49.316 INFO:teuthology.orchestra.run.smithi115.stdout:[mds] 2023-03-25T00:56:49.317 INFO:teuthology.orchestra.run.smithi115.stdout: debug mds = 20 2023-03-25T00:56:49.317 INFO:teuthology.orchestra.run.smithi115.stdout: debug mds balancer = 20 2023-03-25T00:56:49.317 INFO:teuthology.orchestra.run.smithi115.stdout: debug ms = 1 2023-03-25T00:56:49.317 INFO:teuthology.orchestra.run.smithi115.stdout: mds debug frag = True 2023-03-25T00:56:49.317 INFO:teuthology.orchestra.run.smithi115.stdout: mds debug scatterstat = True 2023-03-25T00:56:49.317 INFO:teuthology.orchestra.run.smithi115.stdout: mds op complaint time = 180 2023-03-25T00:56:49.317 INFO:teuthology.orchestra.run.smithi115.stdout: mds verify scatter = True 2023-03-25T00:56:49.318 INFO:teuthology.orchestra.run.smithi115.stdout: osd op complaint time = 180 2023-03-25T00:56:49.318 INFO:teuthology.orchestra.run.smithi115.stdout: rados mon op timeout = 900 2023-03-25T00:56:49.318 INFO:teuthology.orchestra.run.smithi115.stdout: rados osd op timeout = 900 2023-03-25T00:56:49.319 INFO:teuthology.orchestra.run.smithi121.stdout:[global] 2023-03-25T00:56:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: chdir = "" 2023-03-25T00:56:49.319 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-25T00:56:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: auth supported = cephx 2023-03-25T00:56:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: filestore xattr use omap = true 2023-03-25T00:56:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: mon clock drift allowed = 1.000 2023-03-25T00:56:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.320 INFO:teuthology.orchestra.run.smithi121.stdout: osd crush chooseleaf type = 0 2023-03-25T00:56:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: auth debug = true 2023-03-25T00:56:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on old message = true 2023-03-25T00:56:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on bug = true 2023-03-25T00:56:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-25T00:56:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: mon pg warn max object skew = 0 2023-03-25T00:56:49.321 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: # disable pg_autoscaler by default for new pools 2023-03-25T00:56:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-25T00:56:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default size = 2 2023-03-25T00:56:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow primary affinity = true 2023-03-25T00:56:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd allow pg remap = true 2023-03-25T00:56:49.322 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on legacy crush tunables = false 2023-03-25T00:56:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on crush straw calc version zero = false 2023-03-25T00:56:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on no sortbitwise = false 2023-03-25T00:56:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on osd down out interval zero = false 2023-03-25T00:56:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: mon warn on too few osds = false 2023-03-25T00:56:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-25T00:56:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-25T00:56:49.323 INFO:teuthology.orchestra.run.smithi121.stdout: mon_allow_pool_size_one = true 2023-03-25T00:56:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-25T00:56:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: osd default data pool replay window = 5 2023-03-25T00:56:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: mon allow pool delete = true 2023-03-25T00:56:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.324 INFO:teuthology.orchestra.run.smithi121.stdout: mon cluster log file level = debug 2023-03-25T00:56:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: debug asserts on shutdown = true 2023-03-25T00:56:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: mon health detail to clog = false 2023-03-25T00:56:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: mon host = "172.21.15.115,172.21.15.121,[v2:172.21.15.121:3301,v1:172.21.15.121:6790]" 2023-03-25T00:56:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: ms die on skipped message = False 2023-03-25T00:56:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: fsid = 9c1ea6b6-7c28-4560-af27-3b583d106388 2023-03-25T00:56:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.325 INFO:teuthology.orchestra.run.smithi121.stdout:[osd] 2023-03-25T00:56:49.325 INFO:teuthology.orchestra.run.smithi121.stdout: osd journal size = 100 2023-03-25T00:56:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub load threshold = 5.0 2023-03-25T00:56:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: osd scrub max interval = 600 2023-03-25T00:56:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: osd recover clone overlap = true 2023-03-25T00:56:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: osd recovery max chunk = 1048576 2023-03-25T00:56:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.326 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug shutdown = true 2023-03-25T00:56:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug op order = false 2023-03-25T00:56:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug verify stray on activate = true 2023-03-25T00:56:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: osd open classes on start = true 2023-03-25T00:56:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug pg log writeout = true 2023-03-25T00:56:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.327 INFO:teuthology.orchestra.run.smithi121.stdout: osd deep scrub update digest min age = 30 2023-03-25T00:56:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: osd map max advance = 10 2023-03-25T00:56:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: journal zero on create = true 2023-03-25T00:56:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: filestore ondisk finisher threads = 3 2023-03-25T00:56:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: filestore apply finisher threads = 3 2023-03-25T00:56:49.328 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: bdev debug aio = true 2023-03-25T00:56:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: osd debug misdirected ops = true 2023-03-25T00:56:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore block size = 96636764160 2023-03-25T00:56:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: bluestore fsck on mount = True 2023-03-25T00:56:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluefs = 20 2023-03-25T00:56:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: debug bluestore = 20 2023-03-25T00:56:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-25T00:56:49.329 INFO:teuthology.orchestra.run.smithi121.stdout: debug osd = 20 2023-03-25T00:56:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: debug rocksdb = 10 2023-03-25T00:56:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd backfillfull_ratio = 0.85 2023-03-25T00:56:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd full ratio = 0.9 2023-03-25T00:56:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd nearfull ratio = 0.8 2023-03-25T00:56:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: osd failsafe full ratio = 0.95 2023-03-25T00:56:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: osd objectstore = bluestore 2023-03-25T00:56:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2023-03-25T00:56:49.330 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.331 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr] 2023-03-25T00:56:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-25T00:56:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: debug mgr = 20 2023-03-25T00:56:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-03-25T00:56:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-03-25T00:56:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-03-25T00:56:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-03-25T00:56:49.331 INFO:teuthology.orchestra.run.smithi121.stdout: mgr/telemetry/nag = false 2023-03-25T00:56:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.332 INFO:teuthology.orchestra.run.smithi121.stdout:[mon] 2023-03-25T00:56:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-25T00:56:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: debug mon = 20 2023-03-25T00:56:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: debug paxos = 20 2023-03-25T00:56:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: debug auth = 20 2023-03-25T00:56:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: mon data avail warn = 5 2023-03-25T00:56:49.332 INFO:teuthology.orchestra.run.smithi121.stdout: mon mgr mkfs grace = 240 2023-03-25T00:56:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min pgs per osd = 4 2023-03-25T00:56:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd reporter subtree level = osd 2023-03-25T00:56:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: mon osd prime pg temp = true 2023-03-25T00:56:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: mon reweight min bytes per osd = 10 2023-03-25T00:56:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-25T00:56:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: auth mon ticket ttl = 660 # 11m 2023-03-25T00:56:49.333 INFO:teuthology.orchestra.run.smithi121.stdout: auth service ticket ttl = 240 # 4m 2023-03-25T00:56:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: # don't complain about insecure global_id in the test suite 2023-03-25T00:56:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-25T00:56:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-25T00:56:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: # 1m isn't quite enough 2023-03-25T00:56:49.334 INFO:teuthology.orchestra.run.smithi121.stdout: mon_down_mkfs_grace = 2m 2023-03-25T00:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: mon_warn_on_filestore_osds = false 2023-03-25T00:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: mon op complaint time = 120 2023-03-25T00:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: 2023-03-25T00:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout:[client] 2023-03-25T00:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: rgw cache enabled = true 2023-03-25T00:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable ops log = true 2023-03-25T00:56:49.335 INFO:teuthology.orchestra.run.smithi121.stdout: rgw enable usage log = true 2023-03-25T00:56:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-25T00:56:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-25T00:56:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: client mount timeout = 600 2023-03-25T00:56:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: debug client = 20 2023-03-25T00:56:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-25T00:56:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: pid file = "" 2023-03-25T00:56:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2023-03-25T00:56:49.336 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2023-03-25T00:56:49.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.a] 2023-03-25T00:56:49.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.b] 2023-03-25T00:56:49.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mon.c] 2023-03-25T00:56:49.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-03-25T00:56:49.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-03-25T00:56:49.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-03-25T00:56:49.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2023-03-25T00:56:49.337 INFO:teuthology.orchestra.run.smithi121.stdout:[mds] 2023-03-25T00:56:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds = 20 2023-03-25T00:56:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: debug mds balancer = 20 2023-03-25T00:56:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: debug ms = 1 2023-03-25T00:56:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug frag = True 2023-03-25T00:56:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: mds debug scatterstat = True 2023-03-25T00:56:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: mds op complaint time = 180 2023-03-25T00:56:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: mds verify scatter = True 2023-03-25T00:56:49.338 INFO:teuthology.orchestra.run.smithi121.stdout: osd op complaint time = 180 2023-03-25T00:56:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: rados mon op timeout = 900 2023-03-25T00:56:49.339 INFO:teuthology.orchestra.run.smithi121.stdout: rados osd op timeout = 900 2023-03-25T00:56:49.339 INFO:tasks.ceph:Creating admin key on mon.a... 2023-03-25T00:56:49.339 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-03-25T00:56:49.417 INFO:tasks.ceph:Copying monmap to all nodes... 2023-03-25T00:56:49.417 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:49.417 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-03-25T00:56:49.433 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:49.434 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-03-25T00:56:49.490 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi115.front.sepia.ceph.com 2023-03-25T00:56:49.490 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:49.490 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-25T00:56:49.491 DEBUG:teuthology.orchestra.run.smithi115:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-25T00:56:49.565 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:56:49.566 DEBUG:teuthology.orchestra.run.smithi115:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:56:49.622 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi121.front.sepia.ceph.com 2023-03-25T00:56:49.622 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:56:49.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-25T00:56:49.623 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-25T00:56:49.660 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:56:49.660 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:56:49.716 INFO:tasks.ceph:Setting up mon nodes... 2023-03-25T00:56:49.717 INFO:tasks.ceph:Setting up mgr nodes... 2023-03-25T00:56:49.717 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-03-25T00:56:49.770 INFO:teuthology.orchestra.run.smithi115.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-03-25T00:56:49.772 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-03-25T00:56:49.840 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-03-25T00:56:49.842 INFO:tasks.ceph:Setting up mds nodes... 2023-03-25T00:56:49.843 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-03-25T00:56:49.896 INFO:teuthology.orchestra.run.smithi115.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-03-25T00:56:49.898 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-03-25T00:56:49.924 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-03-25T00:56:50.017 INFO:teuthology.orchestra.run.smithi115.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-03-25T00:56:50.019 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-03-25T00:56:50.045 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-03-25T00:56:50.099 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-03-25T00:56:50.101 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-03-25T00:56:50.127 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-03-25T00:56:50.220 INFO:teuthology.orchestra.run.smithi121.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-03-25T00:56:50.223 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-03-25T00:56:50.248 INFO:tasks.ceph_client:Setting up client nodes... 2023-03-25T00:56:50.249 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-03-25T00:56:50.292 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-03-25T00:56:50.305 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-03-25T00:56:50.387 INFO:teuthology.orchestra.run.smithi115.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-03-25T00:56:50.400 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-03-25T00:56:50.443 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-03-25T00:56:50.456 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-03-25T00:56:50.499 INFO:teuthology.orchestra.run.smithi121.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-03-25T00:56:50.511 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-03-25T00:56:50.511 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi115.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi121.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-25T00:56:50.512 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-03-25T00:56:50.537 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-25T00:56:50.538 INFO:tasks.ceph:role: osd.0 2023-03-25T00:56:50.538 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com 2023-03-25T00:56:50.538 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-25T00:56:50.611 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-25T00:56:50.612 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-25T00:56:50.612 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-25T00:56:50.612 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-25T00:56:50.612 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-25T00:56:50.612 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-03-25T00:56:50.612 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-25T00:56:50.612 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-25T00:56:50.613 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-25T00:56:50.613 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-25T00:56:51.053 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-03-25T00:56:51.055 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-03-25T00:56:51.055 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-03-25T00:56:51.219 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-03-25T00:56:51.255 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-03-25T00:56:51.323 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-25T00:56:51.323 INFO:tasks.ceph:role: osd.1 2023-03-25T00:56:51.323 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com 2023-03-25T00:56:51.323 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-25T00:56:51.398 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-25T00:56:51.398 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-25T00:56:51.398 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-25T00:56:51.398 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-25T00:56:51.399 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-25T00:56:51.399 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-03-25T00:56:51.399 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-25T00:56:51.399 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-25T00:56:51.399 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-25T00:56:51.399 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-25T00:56:51.816 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-03-25T00:56:51.818 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-03-25T00:56:51.818 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-03-25T00:56:51.927 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-03-25T00:56:51.958 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-03-25T00:56:52.026 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-25T00:56:52.026 INFO:tasks.ceph:role: osd.2 2023-03-25T00:56:52.027 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com 2023-03-25T00:56:52.027 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-25T00:56:52.100 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-25T00:56:52.101 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-25T00:56:52.101 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-25T00:56:52.101 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-25T00:56:52.101 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-25T00:56:52.101 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-03-25T00:56:52.101 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-25T00:56:52.102 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-25T00:56:52.102 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-25T00:56:52.102 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-25T00:56:52.547 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-03-25T00:56:52.548 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-03-25T00:56:52.549 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-03-25T00:56:52.718 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-03-25T00:56:52.754 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-03-25T00:56:52.824 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-25T00:56:52.824 INFO:tasks.ceph:role: osd.3 2023-03-25T00:56:52.824 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi115.front.sepia.ceph.com 2023-03-25T00:56:52.825 DEBUG:teuthology.orchestra.run.smithi115:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-25T00:56:52.899 INFO:teuthology.orchestra.run.smithi115.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-25T00:56:52.900 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-25T00:56:52.900 INFO:teuthology.orchestra.run.smithi115.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-25T00:56:52.900 INFO:teuthology.orchestra.run.smithi115.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-25T00:56:52.900 INFO:teuthology.orchestra.run.smithi115.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-25T00:56:52.900 INFO:teuthology.orchestra.run.smithi115.stdout: = sunit=0 swidth=0 blks 2023-03-25T00:56:52.900 INFO:teuthology.orchestra.run.smithi115.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-25T00:56:52.901 INFO:teuthology.orchestra.run.smithi115.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-25T00:56:52.901 INFO:teuthology.orchestra.run.smithi115.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-25T00:56:52.901 INFO:teuthology.orchestra.run.smithi115.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-25T00:56:53.346 INFO:teuthology.orchestra.run.smithi115.stdout:Discarding blocks...Done. 2023-03-25T00:56:53.348 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi115.front.sepia.ceph.com -o noatime 2023-03-25T00:56:53.349 DEBUG:teuthology.orchestra.run.smithi115:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-03-25T00:56:53.539 DEBUG:teuthology.orchestra.run.smithi115:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-03-25T00:56:53.572 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:56:53.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:53.679+0000 7ff0e04b8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-03-25T00:56:53.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:53.679+0000 7ff0e04b8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-03-25T00:56:53.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:53.679+0000 7ff0e04b8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-25T00:56:53.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:53.679+0000 7ff0e04b8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-25T00:56:53.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:53.679+0000 7ff0e04b8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-25T00:56:53.683 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:53.680+0000 7ff0e04b8640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-03-25T00:56:56.548 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-25T00:56:56.577 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:56:56.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:56.665+0000 7ff8a2f50640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-03-25T00:56:56.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:56.665+0000 7ff8a2f50640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-03-25T00:56:56.667 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:56.665+0000 7ff8a2f50640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-25T00:56:56.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:56.665+0000 7ff8a2f50640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-25T00:56:56.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:56.665+0000 7ff8a2f50640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-25T00:56:56.668 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:56.666+0000 7ff8a2f50640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-03-25T00:56:59.531 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-25T00:56:59.562 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:56:59.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:59.651+0000 7f41ad178640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-03-25T00:56:59.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:59.651+0000 7f41ad178640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-03-25T00:56:59.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:59.651+0000 7f41ad178640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-25T00:56:59.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:59.651+0000 7f41ad178640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-25T00:56:59.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:59.651+0000 7f41ad178640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-25T00:56:59.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:56:59.652+0000 7f41ad178640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-03-25T00:57:02.518 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-25T00:57:02.550 DEBUG:teuthology.orchestra.run.smithi115:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:57:02.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:02.638+0000 7f75e46b9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-03-25T00:57:02.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:02.639+0000 7f75e46b9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-03-25T00:57:02.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:02.639+0000 7f75e46b9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-25T00:57:02.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:02.639+0000 7f75e46b9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-25T00:57:02.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:02.639+0000 7f75e46b9640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-25T00:57:02.642 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:02.639+0000 7f75e46b9640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-03-25T00:57:05.503 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-25T00:57:05.534 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-03-25T00:57:05.568 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-25T00:57:05.568 INFO:tasks.ceph:role: osd.4 2023-03-25T00:57:05.568 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com 2023-03-25T00:57:05.569 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-25T00:57:05.646 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-25T00:57:05.646 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-25T00:57:05.646 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-25T00:57:05.647 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-25T00:57:05.647 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-25T00:57:05.647 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-03-25T00:57:05.647 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-25T00:57:05.647 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-25T00:57:05.647 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-25T00:57:05.647 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-25T00:57:06.082 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-03-25T00:57:06.084 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-03-25T00:57:06.084 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-03-25T00:57:06.231 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-03-25T00:57:06.265 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-03-25T00:57:06.332 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-25T00:57:06.333 INFO:tasks.ceph:role: osd.5 2023-03-25T00:57:06.333 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com 2023-03-25T00:57:06.333 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-25T00:57:06.408 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-25T00:57:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-25T00:57:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-25T00:57:06.408 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-25T00:57:06.408 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-25T00:57:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-03-25T00:57:06.409 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-25T00:57:06.409 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-25T00:57:06.409 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-25T00:57:06.409 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-25T00:57:06.836 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-03-25T00:57:06.838 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-03-25T00:57:06.838 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-03-25T00:57:06.942 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-03-25T00:57:06.973 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-03-25T00:57:07.040 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-25T00:57:07.040 INFO:tasks.ceph:role: osd.6 2023-03-25T00:57:07.040 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com 2023-03-25T00:57:07.040 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-25T00:57:07.114 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-25T00:57:07.114 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-25T00:57:07.114 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-25T00:57:07.114 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-25T00:57:07.115 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-25T00:57:07.115 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-03-25T00:57:07.115 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-25T00:57:07.115 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-25T00:57:07.115 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-25T00:57:07.115 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-25T00:57:07.559 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-03-25T00:57:07.561 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-03-25T00:57:07.561 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-03-25T00:57:07.747 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-03-25T00:57:07.778 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-03-25T00:57:07.845 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-25T00:57:07.845 INFO:tasks.ceph:role: osd.7 2023-03-25T00:57:07.845 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com 2023-03-25T00:57:07.846 DEBUG:teuthology.orchestra.run.smithi121:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-25T00:57:07.918 INFO:teuthology.orchestra.run.smithi121.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-25T00:57:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-25T00:57:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-25T00:57:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-25T00:57:07.919 INFO:teuthology.orchestra.run.smithi121.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-25T00:57:07.919 INFO:teuthology.orchestra.run.smithi121.stdout: = sunit=0 swidth=0 blks 2023-03-25T00:57:07.920 INFO:teuthology.orchestra.run.smithi121.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-25T00:57:07.920 INFO:teuthology.orchestra.run.smithi121.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-25T00:57:07.920 INFO:teuthology.orchestra.run.smithi121.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-25T00:57:07.920 INFO:teuthology.orchestra.run.smithi121.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-25T00:57:08.367 INFO:teuthology.orchestra.run.smithi121.stdout:Discarding blocks...Done. 2023-03-25T00:57:08.369 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi121.front.sepia.ceph.com -o noatime 2023-03-25T00:57:08.370 DEBUG:teuthology.orchestra.run.smithi121:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-03-25T00:57:08.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-03-25T00:57:08.580 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:57:08.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:08.677+0000 7f2ca2f37640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-03-25T00:57:08.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:08.677+0000 7f2ca2f37640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-03-25T00:57:08.680 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:08.677+0000 7f2ca2f37640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-25T00:57:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:08.677+0000 7f2ca2f37640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-25T00:57:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:08.677+0000 7f2ca2f37640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-25T00:57:08.681 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:08.678+0000 7f2ca2f37640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-03-25T00:57:11.551 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-03-25T00:57:11.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:57:11.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:11.674+0000 7f213a5b0640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-03-25T00:57:11.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:11.674+0000 7f213a5b0640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-03-25T00:57:11.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:11.674+0000 7f213a5b0640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-25T00:57:11.676 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:11.674+0000 7f213a5b0640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-25T00:57:11.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:11.674+0000 7f213a5b0640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-25T00:57:11.677 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:11.675+0000 7f213a5b0640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-03-25T00:57:14.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-03-25T00:57:14.579 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:57:14.672 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:14.670+0000 7f4aa2f19640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-03-25T00:57:14.672 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:14.671+0000 7f4aa2f19640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-03-25T00:57:14.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:14.671+0000 7f4aa2f19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-25T00:57:14.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:14.671+0000 7f4aa2f19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-25T00:57:14.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:14.671+0000 7f4aa2f19640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-25T00:57:14.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:14.671+0000 7f4aa2f19640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-03-25T00:57:17.272 DEBUG:teuthology.orchestra.run.smithi115:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-25T00:57:17.275 DEBUG:teuthology.orchestra.run.smithi121:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-25T00:57:17.550 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-03-25T00:57:17.581 DEBUG:teuthology.orchestra.run.smithi121:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:57:17.672 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:17.670+0000 7f3cf575b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-03-25T00:57:17.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:17.671+0000 7f3cf575b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-03-25T00:57:17.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:17.671+0000 7f3cf575b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-25T00:57:17.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:17.671+0000 7f3cf575b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-25T00:57:17.673 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:17.671+0000 7f3cf575b640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-25T00:57:17.674 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:17.671+0000 7f3cf575b640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-03-25T00:57:20.530 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-25T00:57:20.561 INFO:tasks.ceph:Reading keys from all nodes... 2023-03-25T00:57:20.562 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:20.562 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-03-25T00:57:20.593 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:20.594 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-03-25T00:57:20.664 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:20.664 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-03-25T00:57:20.733 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:20.733 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-03-25T00:57:20.801 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:20.801 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-03-25T00:57:20.869 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:20.869 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-03-25T00:57:20.935 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:20.936 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-03-25T00:57:21.002 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.003 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-03-25T00:57:21.035 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.036 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-03-25T00:57:21.105 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.105 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-03-25T00:57:21.174 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.174 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-03-25T00:57:21.242 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.242 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-03-25T00:57:21.310 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.310 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-03-25T00:57:21.377 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.378 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-03-25T00:57:21.444 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:21.444 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-03-25T00:57:21.462 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:21.463 DEBUG:teuthology.orchestra.run.smithi115:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-03-25T00:57:21.520 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.520 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-03-25T00:57:21.537 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:21.537 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-03-25T00:57:21.594 INFO:tasks.ceph:Adding keys to all mons... 2023-03-25T00:57:21.594 DEBUG:teuthology.orchestra.run.smithi115:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-25T00:57:21.596 DEBUG:teuthology.orchestra.run.smithi121:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-25T00:57:21.637 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.y] 2023-03-25T00:57:21.637 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDRRh5kdnHZLRAAYEBGtoojzskvANFUWFf/Kw== 2023-03-25T00:57:21.638 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.a] 2023-03-25T00:57:21.638 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDRRh5k1q9hNRAAficfAy9Rcar6qu1PjNt8Xw== 2023-03-25T00:57:21.638 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.c] 2023-03-25T00:57:21.638 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDSRh5kGkv1ABAAPDsG1/pnKKJUkVupdzb/eQ== 2023-03-25T00:57:21.638 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.0] 2023-03-25T00:57:21.638 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDVRh5k84yQKBAAEoVtAM4A41KQQuwB7403rA== 2023-03-25T00:57:21.638 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.1] 2023-03-25T00:57:21.639 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDYRh5k2h+2JxAAWF8V4gpKgIgy/kcYPIyXWA== 2023-03-25T00:57:21.639 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.2] 2023-03-25T00:57:21.639 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDbRh5ky5HhJhAA5LT+JpdTRlaLZjXTnk4OWw== 2023-03-25T00:57:21.639 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.3] 2023-03-25T00:57:21.639 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDeRh5kjUMjJhAA3Je7JvZb2uPgQjS8W8ropQ== 2023-03-25T00:57:21.639 INFO:teuthology.orchestra.run.smithi115.stdout:[mgr.x] 2023-03-25T00:57:21.639 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDRRh5kZcQFMhAAfvz5dl+hrqEtgUK5bH5SjQ== 2023-03-25T00:57:21.639 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.b] 2023-03-25T00:57:21.640 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDSRh5kzFDaBRAAx2kMbfGPAuVcAc8jeM4xqg== 2023-03-25T00:57:21.640 INFO:teuthology.orchestra.run.smithi115.stdout:[mds.d] 2023-03-25T00:57:21.640 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDSRh5k4dwZDRAA81Kk+R3h7WoU/Sgt5I6jlA== 2023-03-25T00:57:21.641 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.4] 2023-03-25T00:57:21.641 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDkRh5kvXh7KBAA9ZuOnxGqdCKjLUKkgbQXoA== 2023-03-25T00:57:21.641 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.5] 2023-03-25T00:57:21.641 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDnRh5kl4A7KBAASp8jVwXimrL8zrdE3dsEFA== 2023-03-25T00:57:21.642 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.6] 2023-03-25T00:57:21.642 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDqRh5kjgMEKBAAsLl8ovUmTYYg4WbzWJXmVQ== 2023-03-25T00:57:21.642 INFO:teuthology.orchestra.run.smithi115.stdout:[osd.7] 2023-03-25T00:57:21.642 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDtRh5kGooDKBAA2OSMNrus09WtDEIBWBKCWA== 2023-03-25T00:57:21.642 INFO:teuthology.orchestra.run.smithi115.stdout:[client.0] 2023-03-25T00:57:21.642 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDSRh5kM1FgERAACII4t1aI+6csCRDe602QKw== 2023-03-25T00:57:21.642 INFO:teuthology.orchestra.run.smithi115.stdout:[client.2] 2023-03-25T00:57:21.642 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDSRh5ktx8QFxAAmWEk+MWENT+lmOWJcfIayw== 2023-03-25T00:57:21.643 INFO:teuthology.orchestra.run.smithi115.stdout:[client.1] 2023-03-25T00:57:21.643 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDSRh5k+WtmGhAA5CbVDeKjPuA9UCW5UAkx8A== 2023-03-25T00:57:21.643 INFO:teuthology.orchestra.run.smithi115.stdout:[client.3] 2023-03-25T00:57:21.643 INFO:teuthology.orchestra.run.smithi115.stdout: key = AQDSRh5k5UayHRAA9iSeyMnPB78XG+Ar07CHAQ== 2023-03-25T00:57:21.659 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.y] 2023-03-25T00:57:21.660 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDRRh5kdnHZLRAAYEBGtoojzskvANFUWFf/Kw== 2023-03-25T00:57:21.660 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.a] 2023-03-25T00:57:21.660 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDRRh5k1q9hNRAAficfAy9Rcar6qu1PjNt8Xw== 2023-03-25T00:57:21.660 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.c] 2023-03-25T00:57:21.660 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDSRh5kGkv1ABAAPDsG1/pnKKJUkVupdzb/eQ== 2023-03-25T00:57:21.660 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.0] 2023-03-25T00:57:21.660 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDVRh5k84yQKBAAEoVtAM4A41KQQuwB7403rA== 2023-03-25T00:57:21.661 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.1] 2023-03-25T00:57:21.661 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDYRh5k2h+2JxAAWF8V4gpKgIgy/kcYPIyXWA== 2023-03-25T00:57:21.661 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.2] 2023-03-25T00:57:21.661 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDbRh5ky5HhJhAA5LT+JpdTRlaLZjXTnk4OWw== 2023-03-25T00:57:21.661 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.3] 2023-03-25T00:57:21.661 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDeRh5kjUMjJhAA3Je7JvZb2uPgQjS8W8ropQ== 2023-03-25T00:57:21.661 INFO:teuthology.orchestra.run.smithi121.stdout:[mgr.x] 2023-03-25T00:57:21.661 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDRRh5kZcQFMhAAfvz5dl+hrqEtgUK5bH5SjQ== 2023-03-25T00:57:21.662 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.b] 2023-03-25T00:57:21.662 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDSRh5kzFDaBRAAx2kMbfGPAuVcAc8jeM4xqg== 2023-03-25T00:57:21.662 INFO:teuthology.orchestra.run.smithi121.stdout:[mds.d] 2023-03-25T00:57:21.662 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDSRh5k4dwZDRAA81Kk+R3h7WoU/Sgt5I6jlA== 2023-03-25T00:57:21.662 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.4] 2023-03-25T00:57:21.662 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDkRh5kvXh7KBAA9ZuOnxGqdCKjLUKkgbQXoA== 2023-03-25T00:57:21.662 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.5] 2023-03-25T00:57:21.663 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDnRh5kl4A7KBAASp8jVwXimrL8zrdE3dsEFA== 2023-03-25T00:57:21.663 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.6] 2023-03-25T00:57:21.663 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDqRh5kjgMEKBAAsLl8ovUmTYYg4WbzWJXmVQ== 2023-03-25T00:57:21.663 INFO:teuthology.orchestra.run.smithi121.stdout:[osd.7] 2023-03-25T00:57:21.663 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDtRh5kGooDKBAA2OSMNrus09WtDEIBWBKCWA== 2023-03-25T00:57:21.663 INFO:teuthology.orchestra.run.smithi121.stdout:[client.0] 2023-03-25T00:57:21.663 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDSRh5kM1FgERAACII4t1aI+6csCRDe602QKw== 2023-03-25T00:57:21.664 INFO:teuthology.orchestra.run.smithi121.stdout:[client.2] 2023-03-25T00:57:21.664 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDSRh5ktx8QFxAAmWEk+MWENT+lmOWJcfIayw== 2023-03-25T00:57:21.664 INFO:teuthology.orchestra.run.smithi121.stdout:[client.1] 2023-03-25T00:57:21.664 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDSRh5k+WtmGhAA5CbVDeKjPuA9UCW5UAkx8A== 2023-03-25T00:57:21.664 INFO:teuthology.orchestra.run.smithi121.stdout:[client.3] 2023-03-25T00:57:21.664 INFO:teuthology.orchestra.run.smithi121.stdout: key = AQDSRh5k5UayHRAA9iSeyMnPB78XG+Ar07CHAQ== 2023-03-25T00:57:21.665 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-25T00:57:21.686 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-25T00:57:21.758 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-25T00:57:21.761 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-25T00:57:21.816 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-25T00:57:21.818 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-25T00:57:21.874 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:21.878 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:21.930 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:21.932 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:21.985 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:21.987 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.042 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.044 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.096 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-25T00:57:22.099 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-25T00:57:22.151 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-25T00:57:22.153 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-25T00:57:22.208 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-25T00:57:22.210 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-25T00:57:22.266 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.269 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.321 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.324 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.378 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.380 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.433 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.435 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-25T00:57:22.488 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-25T00:57:22.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-25T00:57:22.546 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-25T00:57:22.548 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-25T00:57:22.604 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-25T00:57:22.606 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-25T00:57:22.662 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-25T00:57:22.664 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-25T00:57:22.720 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-03-25T00:57:22.720 DEBUG:teuthology.orchestra.run.smithi115:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-03-25T00:57:22.746 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-25T00:57:22.844 DEBUG:teuthology.orchestra.run.smithi115:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-03-25T00:57:22.870 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-03-25T00:57:22.897 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-25T00:57:22.994 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-03-25T00:57:23.021 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-03-25T00:57:23.088 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-25T00:57:23.185 DEBUG:teuthology.orchestra.run.smithi121:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-03-25T00:57:23.212 DEBUG:teuthology.orchestra.run.smithi115:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:57:23.214 DEBUG:teuthology.orchestra.run.smithi121:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-25T00:57:23.269 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-03-25T00:57:23.270 INFO:tasks.ceph.mon.a:Restarting daemon 2023-03-25T00:57:23.270 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-03-25T00:57:23.272 INFO:tasks.ceph.mon.a:Started 2023-03-25T00:57:23.272 INFO:tasks.ceph.mon.b:Restarting daemon 2023-03-25T00:57:23.273 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-03-25T00:57:23.311 INFO:tasks.ceph.mon.b:Started 2023-03-25T00:57:23.312 INFO:tasks.ceph.mon.c:Restarting daemon 2023-03-25T00:57:23.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-03-25T00:57:23.314 INFO:tasks.ceph.mon.c:Started 2023-03-25T00:57:23.314 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-03-25T00:57:23.314 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-03-25T00:57:23.315 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-03-25T00:57:23.316 INFO:tasks.ceph.mgr.y:Started 2023-03-25T00:57:23.317 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-03-25T00:57:23.317 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-03-25T00:57:23.319 INFO:tasks.ceph.mgr.x:Started 2023-03-25T00:57:23.319 INFO:tasks.ceph:Setting crush tunables to default 2023-03-25T00:57:23.319 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd crush tunables default 2023-03-25T00:57:23.439 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.437+0000 7f8941f5f700 1 Processor -- start 2023-03-25T00:57:23.440 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.437+0000 7f8941f5f700 1 -- start start 2023-03-25T00:57:23.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f8941f5f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f893c15a9e0 0x7f893c0b3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:23.441 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f8941f5f700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 0x7f893c0c7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:23.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f8941f5f700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893c0b3f90 con 0x7f893c0cb8e0 2023-03-25T00:57:23.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f8941f5f700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893c0c6f00 con 0x7f893c0716f0 2023-03-25T00:57:23.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f8941f5f700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893c15adb0 con 0x7f893c15a9e0 2023-03-25T00:57:23.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f893bfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 0x7f893c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:23.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f893affd700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f893c0cb8e0 0x7f893c070fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:32948/0 (socket says 172.21.15.115:32948) 2023-03-25T00:57:23.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f893bfff700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 0x7f893c0c7350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60624/0 (socket says 172.21.15.115:60624) 2023-03-25T00:57:23.442 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.438+0000 7f893affd700 1 -- 172.21.15.115:0/1833006046 learned_addr learned my addr 172.21.15.115:0/1833006046 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:23.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893bfff700 1 -- 172.21.15.115:0/1833006046 >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 msgr2=0x7f893c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-03-25T00:57:23.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893bfff700 1 -- 172.21.15.115:0/1833006046 >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 msgr2=0x7f893c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-25T00:57:23.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893bfff700 1 --2- 172.21.15.115:0/1833006046 >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 0x7f893c0c7350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-25T00:57:23.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893b7fe700 1 --2- 172.21.15.115:0/1833006046 >> v2:172.21.15.121:3300/0 conn(0x7f893c15a9e0 0x7f893c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:23.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893bfff700 1 --2- 172.21.15.115:0/1833006046 >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 0x7f893c0c7350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-25T00:57:23.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893b7fe700 1 -- 172.21.15.115:0/1833006046 >> v2:172.21.15.121:3300/0 conn(0x7f893c15a9e0 msgr2=0x7f893c0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-03-25T00:57:23.443 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893b7fe700 1 -- 172.21.15.115:0/1833006046 >> v2:172.21.15.121:3300/0 conn(0x7f893c15a9e0 msgr2=0x7f893c0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-25T00:57:23.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893b7fe700 1 --2- 172.21.15.115:0/1833006046 >> v2:172.21.15.121:3300/0 conn(0x7f893c15a9e0 0x7f893c0b3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-25T00:57:23.444 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.439+0000 7f893b7fe700 1 --2- 172.21.15.115:0/1833006046 >> v2:172.21.15.121:3300/0 conn(0x7f893c15a9e0 0x7f893c0b3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-25T00:57:23.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.639+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202774661 0 0) 0x7f893c0b3f90 con 0x7f893c0cb8e0 2023-03-25T00:57:23.641 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.639+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89280035b0 con 0x7f893c0cb8e0 2023-03-25T00:57:23.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1520754851 0 0) 0x7f89280035b0 con 0x7f893c0cb8e0 2023-03-25T00:57:23.643 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 >> v2:172.21.15.121:3300/0 conn(0x7f893c15a9e0 msgr2=0x7f893c0b3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:23.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 --2- 172.21.15.115:0/1833006046 >> v2:172.21.15.121:3300/0 conn(0x7f893c15a9e0 0x7f893c0b3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 msgr2=0x7f893c0c7350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:23.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 --2- 172.21.15.115:0/1833006046 >> v2:172.21.15.115:3300/0 conn(0x7f893c0716f0 0x7f893c0c7350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c0b4270 con 0x7f893c0cb8e0 2023-03-25T00:57:23.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2816979759 0 0) 0x7f8924001670 con 0x7f893c0cb8e0 2023-03-25T00:57:23.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 >> v1:172.21.15.115:6789/0 conn(0x7f893c0cb8e0 legacy=0x7f893c070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:23.644 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928003a50 0x7f8928003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:23.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8928004850 0x7f8928004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:23.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8928005200 0x7f892800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:23.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0c6f00 con 0x7f8928005200 2023-03-25T00:57:23.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c15adb0 con 0x7f8928003a50 2023-03-25T00:57:23.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892800c110 con 0x7f8928004850 2023-03-25T00:57:23.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893bfff700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8928005200 0x7f892800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:23.645 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893b7fe700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928003a50 0x7f8928003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:23.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893affd700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8928004850 0x7f8928004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:23.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893bfff700 1 -- 172.21.15.115:0/1833006046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8928004850 msgr2=0x7f8928004cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:23.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893bfff700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8928004850 0x7f8928004cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893bfff700 1 -- 172.21.15.115:0/1833006046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928003a50 msgr2=0x7f8928003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:23.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893bfff700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928003a50 0x7f8928003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893bfff700 1 -- 172.21.15.115:0/1833006046 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89280035b0 con 0x7f8928005200 2023-03-25T00:57:23.646 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893bfff700 1 -- 172.21.15.115:0/1833006046 reap_dead start 2023-03-25T00:57:23.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893bfff700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8928005200 0x7f892800ba10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f893c06efb0 tx=0x7f8928005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8cba0229dd20475 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:23.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.640+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934003a20 con 0x7f8928005200 2023-03-25T00:57:23.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8934003b80 con 0x7f8928005200 2023-03-25T00:57:23.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f893a7fc700 1 -- 172.21.15.115:0/1833006046 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8934004030 con 0x7f8928005200 2023-03-25T00:57:23.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f8941f5f700 1 -- 172.21.15.115:0/1833006046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8928005200 msgr2=0x7f892800ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:23.647 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f8941f5f700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8928005200 0x7f892800ba10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f893c06efb0 tx=0x7f8928005880 comp rx=0 tx=0).stop 2023-03-25T00:57:23.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f8941f5f700 1 -- 172.21.15.115:0/1833006046 shutdown_connections 2023-03-25T00:57:23.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f8941f5f700 1 --2- 172.21.15.115:0/1833006046 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8928005200 0x7f892800ba10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f8941f5f700 1 -- 172.21.15.115:0/1833006046 >> 172.21.15.115:0/1833006046 conn(0x7f893c0d0b50 msgr2=0x7f893c0cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:23.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f8941f5f700 1 -- 172.21.15.115:0/1833006046 shutdown_connections 2023-03-25T00:57:23.648 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.641+0000 7f8941f5f700 1 -- 172.21.15.115:0/1833006046 wait complete. 2023-03-25T00:57:23.649 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.643+0000 7f8941f5f700 1 Processor -- start 2023-03-25T00:57:23.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.643+0000 7f8941f5f700 1 -- start start 2023-03-25T00:57:23.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f8941f5f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928005200 0x7f893c0cda00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:23.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f8941f5f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f893c14da70 0x7f893c14dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:23.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f8941f5f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f893c1502d0 0x7f893c150770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:23.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f8941f5f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c14e420 con 0x7f893c1502d0 2023-03-25T00:57:23.650 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f8941f5f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0ce100 con 0x7f8928005200 2023-03-25T00:57:23.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f8941f5f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0ce3e0 con 0x7f893c14da70 2023-03-25T00:57:23.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f893c1502d0 0x7f893c150770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:23.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928005200 0x7f893c0cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:23.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893bfff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f893c1502d0 0x7f893c150770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60660/0 (socket says 172.21.15.115:60660) 2023-03-25T00:57:23.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893bfff700 1 -- 172.21.15.115:0/1517106858 learned_addr learned my addr 172.21.15.115:0/1517106858 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:23.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928005200 0x7f893c0cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.115:44954/0 (socket says 172.21.15.115:44954) 2023-03-25T00:57:23.651 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893affd700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f893c14da70 0x7f893c14dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:23.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893b7fe700 1 -- 172.21.15.115:0/1517106858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f893c14da70 msgr2=0x7f893c14dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:23.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893b7fe700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f893c14da70 0x7f893c14dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893b7fe700 1 -- 172.21.15.115:0/1517106858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f893c1502d0 msgr2=0x7f893c150770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:23.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893b7fe700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f893c1502d0 0x7f893c150770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893b7fe700 1 -- 172.21.15.115:0/1517106858 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c0b4830 con 0x7f8928005200 2023-03-25T00:57:23.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893bfff700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f893c1502d0 0x7f893c150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:23.652 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.644+0000 7f893b7fe700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928005200 0x7f893c0cda00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f892c00f600 tx=0x7f892c00f910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:23.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.649+0000 7f8933fff700 1 -- 172.21.15.115:0/1517106858 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c004440 con 0x7f8928005200 2023-03-25T00:57:23.653 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.649+0000 7f8933fff700 1 -- 172.21.15.115:0/1517106858 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892c00c070 con 0x7f8928005200 2023-03-25T00:57:23.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.649+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c0b4550 con 0x7f8928005200 2023-03-25T00:57:23.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.649+0000 7f8933fff700 1 -- 172.21.15.115:0/1517106858 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892c0038e0 con 0x7f8928005200 2023-03-25T00:57:23.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.649+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c0b4b10 con 0x7f8928005200 2023-03-25T00:57:23.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.650+0000 7f8933fff700 1 -- 172.21.15.115:0/1517106858 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f892c003d40 con 0x7f8928005200 2023-03-25T00:57:23.654 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.650+0000 7f8933fff700 1 -- 172.21.15.115:0/1517106858 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f892c01cce0 con 0x7f8928005200 2023-03-25T00:57:23.655 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.650+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893c158150 con 0x7f8928005200 2023-03-25T00:57:23.659 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.657+0000 7f8933fff700 1 -- 172.21.15.115:0/1517106858 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f892c03bb10 con 0x7f8928005200 2023-03-25T00:57:23.733 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.731+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f893c157e70 con 0x7f8928005200 2023-03-25T00:57:23.738 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.736+0000 7f8933fff700 1 -- 172.21.15.115:0/1517106858 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f892c02a020 con 0x7f8928005200 2023-03-25T00:57:23.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928005200 msgr2=0x7f893c0cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:23.739 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928005200 0x7f893c0cda00 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f892c00f600 tx=0x7f892c00f910 comp rx=0 tx=0).stop 2023-03-25T00:57:23.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 shutdown_connections 2023-03-25T00:57:23.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8928005200 0x7f893c0cda00 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f893c1502d0 0x7f893c150770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 --2- 172.21.15.115:0/1517106858 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f893c14da70 0x7f893c14dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:23.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 >> 172.21.15.115:0/1517106858 conn(0x7f893c0d0b50 msgr2=0x7f893c15c670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:23.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 shutdown_connections 2023-03-25T00:57:23.740 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:23.737+0000 7f8941f5f700 1 -- 172.21.15.115:0/1517106858 wait complete. 2023-03-25T00:57:23.741 INFO:teuthology.orchestra.run.smithi115.stderr:adjusted tunables profile to default 2023-03-25T00:57:23.751 INFO:tasks.ceph:check_enable_crimson: False 2023-03-25T00:57:23.751 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-03-25T00:57:23.752 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:23.752 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-03-25T00:57:23.757 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:23.756+0000 7f9bea30b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-25T00:57:23.758 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:23.756+0000 7fa7398e6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-25T00:57:23.778 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:23.778 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-03-25T00:57:23.822 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:23.821+0000 7f9bea30b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-25T00:57:23.826 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:23.825+0000 7fa7398e6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-25T00:57:23.843 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:23.844 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-03-25T00:57:23.896 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:23.895+0000 7f9bea30b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-25T00:57:23.905 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:23.903+0000 7fa7398e6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-25T00:57:23.910 DEBUG:teuthology.orchestra.run.smithi115:> set -ex 2023-03-25T00:57:23.910 DEBUG:teuthology.orchestra.run.smithi115:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-03-25T00:57:23.974 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:23.973+0000 7f9bea30b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-25T00:57:23.976 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:23.976 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-03-25T00:57:23.988 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:23.987+0000 7fa7398e6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-25T00:57:24.002 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:24.002 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-03-25T00:57:24.069 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:24.069 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-03-25T00:57:24.123 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:24.122+0000 7f9bea30b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-25T00:57:24.131 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:24.129+0000 7fa7398e6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-25T00:57:24.135 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2023-03-25T00:57:24.136 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-03-25T00:57:24.201 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new b6e324a3-c8ad-459c-bb1e-256d32a317db 0 2023-03-25T00:57:24.361 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.360+0000 7f8a8535d700 1 Processor -- start 2023-03-25T00:57:24.362 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.360+0000 7f8a8535d700 1 -- start start 2023-03-25T00:57:24.363 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a8535d700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8a801584f0 0x7f8a800b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a8535d700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f8a800b2fa0 0x7f8a800cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a8535d700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a801588c0 con 0x7f8a8015bc50 2023-03-25T00:57:24.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a8535d700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800b3420 con 0x7f8a800b2fa0 2023-03-25T00:57:24.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a8535d700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a800ccb90 con 0x7f8a801584f0 2023-03-25T00:57:24.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7effd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f8a801584f0 0x7f8a800b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.364 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7e7fc700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f8a8015bc50 0x7f8a800d00c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54476/0 (socket says 172.21.15.121:54476) 2023-03-25T00:57:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7e7fc700 1 -- 172.21.15.121:0/1815189830 learned_addr learned my addr 172.21.15.121:0/1815189830 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-25T00:57:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7f7fe700 1 --2- 172.21.15.121:0/1815189830 >> v2:172.21.15.115:3300/0 conn(0x7f8a800b2fa0 0x7f8a800cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7effd700 1 -- 172.21.15.121:0/1815189830 >> v2:172.21.15.115:3300/0 conn(0x7f8a800b2fa0 msgr2=0x7f8a800cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7effd700 1 --2- 172.21.15.121:0/1815189830 >> v2:172.21.15.115:3300/0 conn(0x7f8a800b2fa0 0x7f8a800cedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7effd700 1 -- 172.21.15.121:0/1815189830 >> v1:172.21.15.121:6789/0 conn(0x7f8a8015bc50 legacy=0x7f8a800d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7dffb700 1 -- 172.21.15.121:0/1815189830 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3932094305 0 0) 0x7f8a801588c0 con 0x7f8a8015bc50 2023-03-25T00:57:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7effd700 1 -- 172.21.15.121:0/1815189830 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a800cce70 con 0x7f8a801584f0 2023-03-25T00:57:24.365 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7effd700 1 --2- 172.21.15.121:0/1815189830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a800b2a60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8a800735c0 tx=0x7f8a70009b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac030580c4b31646 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:24.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7dffb700 1 -- 172.21.15.121:0/1815189830 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70011830 con 0x7f8a801584f0 2023-03-25T00:57:24.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7dffb700 1 -- 172.21.15.121:0/1815189830 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a70011990 con 0x7f8a801584f0 2023-03-25T00:57:24.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.361+0000 7f8a7dffb700 1 -- 172.21.15.121:0/1815189830 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70011d20 con 0x7f8a801584f0 2023-03-25T00:57:24.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.362+0000 7f8a8535d700 1 -- 172.21.15.121:0/1815189830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 msgr2=0x7f8a800b2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.362+0000 7f8a8535d700 1 --2- 172.21.15.121:0/1815189830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a800b2a60 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8a800735c0 tx=0x7f8a70009b20 comp rx=0 tx=0).stop 2023-03-25T00:57:24.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.362+0000 7f8a8535d700 1 -- 172.21.15.121:0/1815189830 shutdown_connections 2023-03-25T00:57:24.366 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.362+0000 7f8a8535d700 1 --2- 172.21.15.121:0/1815189830 >> v2:172.21.15.115:3300/0 conn(0x7f8a800b2fa0 0x7f8a800cedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.362+0000 7f8a8535d700 1 --2- 172.21.15.121:0/1815189830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a800b2a60 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.362+0000 7f8a8535d700 1 -- 172.21.15.121:0/1815189830 >> 172.21.15.121:0/1815189830 conn(0x7f8a800d0fe0 msgr2=0x7f8a800d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:24.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.362+0000 7f8a8535d700 1 -- 172.21.15.121:0/1815189830 shutdown_connections 2023-03-25T00:57:24.367 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.362+0000 7f8a8535d700 1 -- 172.21.15.121:0/1815189830 wait complete. 2023-03-25T00:57:24.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.363+0000 7f8a8535d700 1 Processor -- start 2023-03-25T00:57:24.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.363+0000 7f8a8535d700 1 -- start start 2023-03-25T00:57:24.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a8535d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a8015a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a8535d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a8014d3d0 0x7f8a80159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a8535d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a801596a0 0x7f8a80159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.369 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a8535d700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800b1450 con 0x7f8a8014d3d0 2023-03-25T00:57:24.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a8535d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800b1fd0 con 0x7f8a801584f0 2023-03-25T00:57:24.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a8535d700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800b16a0 con 0x7f8a801596a0 2023-03-25T00:57:24.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a8015a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a8015a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44962/0 (socket says 172.21.15.121:44962) 2023-03-25T00:57:24.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7effd700 1 -- 172.21.15.121:0/812595847 learned_addr learned my addr 172.21.15.121:0/812595847 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:24.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7f7fe700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a801596a0 0x7f8a80159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7e7fc700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a8014d3d0 0x7f8a80159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.370 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7effd700 1 -- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a801596a0 msgr2=0x7f8a80159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7effd700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a801596a0 0x7f8a80159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7effd700 1 -- 172.21.15.121:0/812595847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a8014d3d0 msgr2=0x7f8a80159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7effd700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a8014d3d0 0x7f8a80159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7effd700 1 -- 172.21.15.121:0/812595847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700094a0 con 0x7f8a801584f0 2023-03-25T00:57:24.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.364+0000 7f8a7f7fe700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a801596a0 0x7f8a80159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:24.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.365+0000 7f8a7effd700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a8015a430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8a700090b0 tx=0x7f8a70004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:24.371 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.365+0000 7f8a677fe700 1 -- 172.21.15.121:0/812595847 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a700036a0 con 0x7f8a801584f0 2023-03-25T00:57:24.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.365+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a800cc7b0 con 0x7f8a801584f0 2023-03-25T00:57:24.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.365+0000 7f8a677fe700 1 -- 172.21.15.121:0/812595847 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7001f070 con 0x7f8a801584f0 2023-03-25T00:57:24.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.365+0000 7f8a677fe700 1 -- 172.21.15.121:0/812595847 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70003a60 con 0x7f8a801584f0 2023-03-25T00:57:24.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.365+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a800cd150 con 0x7f8a801584f0 2023-03-25T00:57:24.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.365+0000 7f8a677fe700 1 -- 172.21.15.121:0/812595847 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f8a70026480 con 0x7f8a801584f0 2023-03-25T00:57:24.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.365+0000 7f8a677fe700 1 -- 172.21.15.121:0/812595847 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8a70021070 con 0x7f8a801584f0 2023-03-25T00:57:24.372 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.366+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a800cc940 con 0x7f8a801584f0 2023-03-25T00:57:24.373 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.368+0000 7f8a677fe700 1 -- 172.21.15.121:0/812595847 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f8a70015020 con 0x7f8a801584f0 2023-03-25T00:57:24.453 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.452+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b6e324a3-c8ad-459c-bb1e-256d32a317db", "id": 0} v 0) v1 -- 0x7f8a80074580 con 0x7f8a801584f0 2023-03-25T00:57:24.458 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.457+0000 7f8a677fe700 1 -- 172.21.15.121:0/812595847 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6e324a3-c8ad-459c-bb1e-256d32a317db", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8a700269f0 con 0x7f8a801584f0 2023-03-25T00:57:24.459 INFO:teuthology.orchestra.run.smithi121.stdout:0 2023-03-25T00:57:24.459 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f9bea30b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-25T00:57:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 msgr2=0x7f8a8015a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a8015a430 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8a700090b0 tx=0x7f8a70004750 comp rx=0 tx=0).stop 2023-03-25T00:57:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 shutdown_connections 2023-03-25T00:57:24.460 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a801584f0 0x7f8a8015a430 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f8a801596a0 0x7f8a80159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 --2- 172.21.15.121:0/812595847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f8a8014d3d0 0x7f8a80159160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 >> 172.21.15.121:0/812595847 conn(0x7f8a800d0fe0 msgr2=0x7f8a8006ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 shutdown_connections 2023-03-25T00:57:24.461 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.458+0000 7f8a8535d700 1 -- 172.21.15.121:0/812595847 wait complete. 2023-03-25T00:57:24.471 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 009b7c98-0c48-49f2-98a1-e8d7afe9af58 1 2023-03-25T00:57:24.483 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:24.482+0000 7fa7398e6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-25T00:57:24.541 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:24.539+0000 7f9bea30b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-25T00:57:24.558 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:24.556+0000 7fa7398e6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-25T00:57:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.593+0000 7f0acfbac700 1 Processor -- start 2023-03-25T00:57:24.595 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.593+0000 7f0acfbac700 1 -- start start 2023-03-25T00:57:24.596 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acfbac700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ac815ca00 0x7f0ac8070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acfbac700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac80b57c0 0x7f0ac8071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acfbac700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac80cda00 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acfbac700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac80cdc50 con 0x7f0ac815ca00 2023-03-25T00:57:24.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acfbac700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac80cdf30 con 0x7f0ac80b57c0 2023-03-25T00:57:24.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acd948700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ac815ca00 0x7f0ac8070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acd147700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac80b57c0 0x7f0ac8071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.597 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acd948700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f0ac815ca00 0x7f0ac8070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44968/0 (socket says 172.21.15.121:44968) 2023-03-25T00:57:24.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acd147700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac80b57c0 0x7f0ac8071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44850/0 (socket says 172.21.15.121:44850) 2023-03-25T00:57:24.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.594+0000 7f0acd948700 1 -- 172.21.15.121:0/856969033 learned_addr learned my addr 172.21.15.121:0/856969033 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:24.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acd948700 1 -- 172.21.15.121:0/856969033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac80b57c0 msgr2=0x7f0ac8071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:24.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acd948700 1 --2- 172.21.15.121:0/856969033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac80b57c0 0x7f0ac8071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acd948700 1 -- 172.21.15.121:0/856969033 >> v1:172.21.15.115:6789/0 conn(0x7f0ac80cc0b0 legacy=0x7f0ac80ca480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acd948700 1 -- 172.21.15.121:0/856969033 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac80ce210 con 0x7f0ac815ca00 2023-03-25T00:57:24.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acd147700 1 --2- 172.21.15.121:0/856969033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac80b57c0 0x7f0ac8071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:24.598 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acd948700 1 --2- 172.21.15.121:0/856969033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac815ca00 0x7f0ac8070fe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40098f0 tx=0x7f0ac4009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e47602906d702a1a server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:24.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acc946700 1 -- 172.21.15.121:0/856969033 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4011830 con 0x7f0ac815ca00 2023-03-25T00:57:24.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acc946700 1 -- 172.21.15.121:0/856969033 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ac4011990 con 0x7f0ac815ca00 2023-03-25T00:57:24.599 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acc946700 1 -- 172.21.15.121:0/856969033 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac4011d20 con 0x7f0ac815ca00 2023-03-25T00:57:24.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acfbac700 1 -- 172.21.15.121:0/856969033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac815ca00 msgr2=0x7f0ac8070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acfbac700 1 --2- 172.21.15.121:0/856969033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac815ca00 0x7f0ac8070fe0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0ac40098f0 tx=0x7f0ac4009c00 comp rx=0 tx=0).stop 2023-03-25T00:57:24.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acfbac700 1 -- 172.21.15.121:0/856969033 shutdown_connections 2023-03-25T00:57:24.601 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acfbac700 1 --2- 172.21.15.121:0/856969033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac815ca00 0x7f0ac8070fe0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acfbac700 1 --2- 172.21.15.121:0/856969033 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac80b57c0 0x7f0ac8071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.595+0000 7f0acfbac700 1 -- 172.21.15.121:0/856969033 >> 172.21.15.121:0/856969033 conn(0x7f0ac80d0b50 msgr2=0x7f0ac80cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.596+0000 7f0acfbac700 1 -- 172.21.15.121:0/856969033 shutdown_connections 2023-03-25T00:57:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.596+0000 7f0acfbac700 1 -- 172.21.15.121:0/856969033 wait complete. 2023-03-25T00:57:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.597+0000 7f0acfbac700 1 Processor -- start 2023-03-25T00:57:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.597+0000 7f0acfbac700 1 -- start start 2023-03-25T00:57:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acfbac700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ac80b57c0 0x7f0ac8154c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.602 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acfbac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac80cc0b0 0x7f0ac80c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acfbac700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac815ca00 0x7f0ac80c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acfbac700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac815e560 con 0x7f0ac80b57c0 2023-03-25T00:57:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acfbac700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac8155150 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acfbac700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac81555d0 con 0x7f0ac815ca00 2023-03-25T00:57:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd147700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac80cc0b0 0x7f0ac80c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd147700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac80cc0b0 0x7f0ac80c5980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44974/0 (socket says 172.21.15.121:44974) 2023-03-25T00:57:24.603 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd147700 1 -- 172.21.15.121:0/3707460532 learned_addr learned my addr 172.21.15.121:0/3707460532 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0ace149700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac815ca00 0x7f0ac80c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd948700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ac80b57c0 0x7f0ac8154c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd147700 1 -- 172.21.15.121:0/3707460532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac815ca00 msgr2=0x7f0ac80c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd147700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac815ca00 0x7f0ac80c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd147700 1 -- 172.21.15.121:0/3707460532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ac80b57c0 msgr2=0x7f0ac8154c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd147700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ac80b57c0 0x7f0ac8154c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.598+0000 7f0acd147700 1 -- 172.21.15.121:0/3707460532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab80094a0 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.604 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.599+0000 7f0acd147700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac80cc0b0 0x7f0ac80c5980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800ec10 tx=0x7f0ab800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.599+0000 7f0abe7fc700 1 -- 172.21.15.121:0/3707460532 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800c850 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.599+0000 7f0abe7fc700 1 -- 172.21.15.121:0/3707460532 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab800c9b0 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.599+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ac4009550 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.599+0000 7f0abe7fc700 1 -- 172.21.15.121:0/3707460532 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800cd20 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.599+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac815cdd0 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.599+0000 7f0abe7fc700 1 -- 172.21.15.121:0/3707460532 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0ab8005d20 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.599+0000 7f0abe7fc700 1 -- 172.21.15.121:0/3707460532 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f0ab8013820 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.605 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.600+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac80ce4f0 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.606 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.602+0000 7f0abe7fc700 1 -- 172.21.15.121:0/3707460532 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f0ab8021060 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.677 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:24.675+0000 7f9bea30b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-25T00:57:24.687 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.686+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "009b7c98-0c48-49f2-98a1-e8d7afe9af58", "id": 1} v 0) v1 -- 0x7f0ac8071d40 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.693 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:24.691+0000 7fa7398e6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-25T00:57:24.695 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.694+0000 7f0abe7fc700 1 -- 172.21.15.121:0/3707460532 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "009b7c98-0c48-49f2-98a1-e8d7afe9af58", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0ab8017070 con 0x7f0ac80cc0b0 2023-03-25T00:57:24.695 INFO:teuthology.orchestra.run.smithi121.stdout:1 2023-03-25T00:57:24.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.694+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac80cc0b0 msgr2=0x7f0ac80c5980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.694+0000 7f0acfbac700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac80cc0b0 0x7f0ac80c5980 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800ec10 tx=0x7f0ab800ef20 comp rx=0 tx=0).stop 2023-03-25T00:57:24.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.694+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 shutdown_connections 2023-03-25T00:57:24.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.694+0000 7f0acfbac700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f0ac815ca00 0x7f0ac80c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.696 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.694+0000 7f0acfbac700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f0ac80b57c0 0x7f0ac8154c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.694+0000 7f0acfbac700 1 --2- 172.21.15.121:0/3707460532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0ac80cc0b0 0x7f0ac80c5980 secure :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800ec10 tx=0x7f0ab800ef20 comp rx=0 tx=0).stop 2023-03-25T00:57:24.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.694+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 >> 172.21.15.121:0/3707460532 conn(0x7f0ac80d0b50 msgr2=0x7f0ac815b430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:24.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.695+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 shutdown_connections 2023-03-25T00:57:24.697 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.695+0000 7f0acfbac700 1 -- 172.21.15.121:0/3707460532 wait complete. 2023-03-25T00:57:24.708 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new b4056d77-ca36-481f-9779-017fb707bda5 2 2023-03-25T00:57:24.765 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:24.764+0000 7f9bea30b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-25T00:57:24.780 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:24.778+0000 7fa7398e6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-25T00:57:24.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.827+0000 7f5739190700 1 Processor -- start 2023-03-25T00:57:24.829 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.827+0000 7f5739190700 1 -- start start 2023-03-25T00:57:24.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f5739190700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f5739190700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f57340cbd60 0x7f5734070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f5739190700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f57340cc1d0 0x7f5734071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f5739190700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5734160610 con 0x7f57340cc1d0 2023-03-25T00:57:24.831 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f5739190700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57341608f0 con 0x7f57340cbd60 2023-03-25T00:57:24.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f5739190700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5734160bd0 con 0x7f57340cf7e0 2023-03-25T00:57:24.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f57340cbd60 0x7f5734070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f5732d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f57340cbd60 0x7f5734070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:44980/0 (socket says 172.21.15.121:44980) 2023-03-25T00:57:24.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 -- 172.21.15.121:0/3239821417 learned_addr learned my addr 172.21.15.121:0/3239821417 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:24.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573359e700 1 --2- 172.21.15.121:0/3239821417 >> v2:172.21.15.115:3300/0 conn(0x7f57340cc1d0 0x7f5734071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.832 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 -- 172.21.15.121:0/3239821417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 msgr2=0x7f57340ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:24.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 --2- 172.21.15.121:0/3239821417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 -- 172.21.15.121:0/3239821417 >> v2:172.21.15.115:3300/0 conn(0x7f57340cc1d0 msgr2=0x7f5734071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:24.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 --2- 172.21.15.121:0/3239821417 >> v2:172.21.15.115:3300/0 conn(0x7f57340cc1d0 0x7f5734071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 -- 172.21.15.121:0/3239821417 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5734160eb0 con 0x7f57340cbd60 2023-03-25T00:57:24.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f5732d9d700 1 --2- 172.21.15.121:0/3239821417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:24.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.828+0000 7f573259c700 1 --2- 172.21.15.121:0/3239821417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cbd60 0x7f5734070fe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5734073e30 tx=0x7f5728009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfe9727a2f133d14 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:24.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5731d9b700 1 -- 172.21.15.121:0/3239821417 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728011830 con 0x7f57340cbd60 2023-03-25T00:57:24.833 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5731d9b700 1 -- 172.21.15.121:0/3239821417 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5728011990 con 0x7f57340cbd60 2023-03-25T00:57:24.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5731d9b700 1 -- 172.21.15.121:0/3239821417 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728011d20 con 0x7f57340cbd60 2023-03-25T00:57:24.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 -- 172.21.15.121:0/3239821417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cbd60 msgr2=0x7f5734070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 --2- 172.21.15.121:0/3239821417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cbd60 0x7f5734070fe0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5734073e30 tx=0x7f5728009bd0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 -- 172.21.15.121:0/3239821417 shutdown_connections 2023-03-25T00:57:24.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 --2- 172.21.15.121:0/3239821417 >> v2:172.21.15.115:3300/0 conn(0x7f57340cc1d0 0x7f5734071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 --2- 172.21.15.121:0/3239821417 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.834 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 --2- 172.21.15.121:0/3239821417 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cbd60 0x7f5734070fe0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 -- 172.21.15.121:0/3239821417 >> 172.21.15.121:0/3239821417 conn(0x7f57340d0b50 msgr2=0x7f57340cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:24.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 -- 172.21.15.121:0/3239821417 shutdown_connections 2023-03-25T00:57:24.835 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.829+0000 7f5739190700 1 -- 172.21.15.121:0/3239821417 wait complete. 2023-03-25T00:57:24.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.831+0000 7f5739190700 1 Processor -- start 2023-03-25T00:57:24.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.831+0000 7f5739190700 1 -- start start 2023-03-25T00:57:24.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.831+0000 7f5739190700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57340cbd60 0x7f5734156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.831+0000 7f5739190700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cc1d0 0x7f57340c9740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.831+0000 7f5739190700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:24.837 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.831+0000 7f5739190700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57340aa0b0 con 0x7f57340cbd60 2023-03-25T00:57:24.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.831+0000 7f5739190700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5734157290 con 0x7f57340cc1d0 2023-03-25T00:57:24.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.831+0000 7f5739190700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57340a9430 con 0x7f57340cf7e0 2023-03-25T00:57:24.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573359e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cc1d0 0x7f57340c9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573359e700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340c9c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44870/0 (socket says 172.21.15.121:44870) 2023-03-25T00:57:24.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573359e700 1 -- 172.21.15.121:0/242713270 learned_addr learned my addr 172.21.15.121:0/242713270 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:24.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f5732d9d700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57340cbd60 0x7f5734156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:24.838 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573259c700 1 -- 172.21.15.121:0/242713270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 msgr2=0x7f57340c9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.839 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573259c700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340c9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.839 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573259c700 1 -- 172.21.15.121:0/242713270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57340cbd60 msgr2=0x7f5734156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.839 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573259c700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57340cbd60 0x7f5734156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.839 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573259c700 1 -- 172.21.15.121:0/242713270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57240094a0 con 0x7f57340cc1d0 2023-03-25T00:57:24.839 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f5732d9d700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57340cbd60 0x7f5734156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:24.839 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573359e700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:24.839 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f573259c700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cc1d0 0x7f57340c9740 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5728016000 tx=0x7f5728004750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:24.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f571b7fe700 1 -- 172.21.15.121:0/242713270 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5728004240 con 0x7f57340cc1d0 2023-03-25T00:57:24.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f571b7fe700 1 -- 172.21.15.121:0/242713270 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f572801f070 con 0x7f57340cc1d0 2023-03-25T00:57:24.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5728009550 con 0x7f57340cc1d0 2023-03-25T00:57:24.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f571b7fe700 1 -- 172.21.15.121:0/242713270 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57280036a0 con 0x7f57340cc1d0 2023-03-25T00:57:24.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5734161470 con 0x7f57340cc1d0 2023-03-25T00:57:24.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f571b7fe700 1 -- 172.21.15.121:0/242713270 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f5728021030 con 0x7f57340cc1d0 2023-03-25T00:57:24.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.832+0000 7f571b7fe700 1 -- 172.21.15.121:0/242713270 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f572801aaa0 con 0x7f57340cc1d0 2023-03-25T00:57:24.840 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.833+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5734161190 con 0x7f57340cc1d0 2023-03-25T00:57:24.841 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.835+0000 7f571b7fe700 1 -- 172.21.15.121:0/242713270 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f5728015020 con 0x7f57340cc1d0 2023-03-25T00:57:24.920 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.919+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b4056d77-ca36-481f-9779-017fb707bda5", "id": 2} v 0) v1 -- 0x7f57340cfbb0 con 0x7f57340cc1d0 2023-03-25T00:57:24.928 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.927+0000 7f571b7fe700 1 -- 172.21.15.121:0/242713270 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b4056d77-ca36-481f-9779-017fb707bda5", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5728003ce0 con 0x7f57340cc1d0 2023-03-25T00:57:24.928 INFO:teuthology.orchestra.run.smithi121.stdout:2 2023-03-25T00:57:24.929 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cc1d0 msgr2=0x7f57340c9740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:24.929 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cc1d0 0x7f57340c9740 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5728016000 tx=0x7f5728004750 comp rx=0 tx=0).stop 2023-03-25T00:57:24.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 shutdown_connections 2023-03-25T00:57:24.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f57340cf7e0 0x7f57340c9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f57340cbd60 0x7f5734156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 --2- 172.21.15.121:0/242713270 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57340cc1d0 0x7f57340c9740 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:24.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 >> 172.21.15.121:0/242713270 conn(0x7f57340d0b50 msgr2=0x7f573415d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:24.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 shutdown_connections 2023-03-25T00:57:24.930 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:24.928+0000 7f5739190700 1 -- 172.21.15.121:0/242713270 wait complete. 2023-03-25T00:57:24.941 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 2c09166f-791d-42cf-9ef4-4746592f74e2 3 2023-03-25T00:57:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.059+0000 7fcad733b700 1 Processor -- start 2023-03-25T00:57:25.061 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.059+0000 7fcad733b700 1 -- start start 2023-03-25T00:57:25.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcad733b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad0160ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.062 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcad733b700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad00cda00 con 0x7fcad015ca00 2023-03-25T00:57:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcad733b700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcad00cdc50 con 0x7fcad00b57c0 2023-03-25T00:57:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcad733b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00cdf30 con 0x7fcad00716f0 2023-03-25T00:57:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcad58d8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad0160ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcad48d6700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7fcad00b57c0 0x7fcad01605d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54510/0 (socket says 172.21.15.121:54510) 2023-03-25T00:57:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcad48d6700 1 -- 172.21.15.121:0/2981665064 learned_addr learned my addr 172.21.15.121:0/2981665064 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-25T00:57:25.063 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcad58d8700 1 -- 172.21.15.121:0/2981665064 >> v1:172.21.15.121:6789/0 conn(0x7fcad00b57c0 legacy=0x7fcad01605d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad58d8700 1 -- 172.21.15.121:0/2981665064 >> v1:172.21.15.115:6789/0 conn(0x7fcad015ca00 legacy=0x7fcad0070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:25.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.060+0000 7fcac7fff700 1 -- 172.21.15.121:0/2981665064 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3370114974 0 0) 0x7fcad00cdc50 con 0x7fcad00b57c0 2023-03-25T00:57:25.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad58d8700 1 -- 172.21.15.121:0/2981665064 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcad00ce210 con 0x7fcad00716f0 2023-03-25T00:57:25.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad58d8700 1 --2- 172.21.15.121:0/2981665064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad0160ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009dc0 tx=0x7fcac800b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72eac27c5963ab9c server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:25.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcac7fff700 1 -- 172.21.15.121:0/2981665064 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac800e070 con 0x7fcad00716f0 2023-03-25T00:57:25.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcac7fff700 1 -- 172.21.15.121:0/2981665064 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac800bbd0 con 0x7fcad00716f0 2023-03-25T00:57:25.064 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcac7fff700 1 -- 172.21.15.121:0/2981665064 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac8012430 con 0x7fcad00716f0 2023-03-25T00:57:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad733b700 1 -- 172.21.15.121:0/2981665064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 msgr2=0x7fcad0160ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad733b700 1 --2- 172.21.15.121:0/2981665064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad0160ce0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcac8009dc0 tx=0x7fcac800b330 comp rx=0 tx=0).stop 2023-03-25T00:57:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad733b700 1 -- 172.21.15.121:0/2981665064 shutdown_connections 2023-03-25T00:57:25.067 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad733b700 1 --2- 172.21.15.121:0/2981665064 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad0160ce0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad733b700 1 -- 172.21.15.121:0/2981665064 >> 172.21.15.121:0/2981665064 conn(0x7fcad00d0b50 msgr2=0x7fcad00cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad733b700 1 -- 172.21.15.121:0/2981665064 shutdown_connections 2023-03-25T00:57:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.061+0000 7fcad733b700 1 -- 172.21.15.121:0/2981665064 wait complete. 2023-03-25T00:57:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad733b700 1 Processor -- start 2023-03-25T00:57:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad733b700 1 -- start start 2023-03-25T00:57:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad733b700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad014dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.068 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad733b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcad00b57c0 0x7fcad00c93f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad733b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcad015ca00 0x7fcad0156b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad733b700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00c3920 con 0x7fcad015ca00 2023-03-25T00:57:25.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad733b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00c3b00 con 0x7fcad00b57c0 2023-03-25T00:57:25.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad733b700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcad00c3de0 con 0x7fcad00716f0 2023-03-25T00:57:25.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad50d7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad014dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad48d6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcad00b57c0 0x7fcad00c93f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.069 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad50d7700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad014dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44878/0 (socket says 172.21.15.121:44878) 2023-03-25T00:57:25.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.063+0000 7fcad50d7700 1 -- 172.21.15.121:0/203105576 learned_addr learned my addr 172.21.15.121:0/203105576 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:25.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad58d8700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcad015ca00 0x7fcad0156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad50d7700 1 -- 172.21.15.121:0/203105576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcad00b57c0 msgr2=0x7fcad00c93f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad50d7700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcad00b57c0 0x7fcad00c93f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad50d7700 1 -- 172.21.15.121:0/203105576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcad015ca00 msgr2=0x7fcad0156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad50d7700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcad015ca00 0x7fcad0156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.070 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad50d7700 1 -- 172.21.15.121:0/203105576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac8009a70 con 0x7fcad00716f0 2023-03-25T00:57:25.071 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad48d6700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcad00b57c0 0x7fcad00c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:25.071 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad58d8700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcad015ca00 0x7fcad0156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-25T00:57:25.071 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad50d7700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad014dca0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcac000c450 tx=0x7fcac000c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:25.071 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcac5ffb700 1 -- 172.21.15.121:0/203105576 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac00052d0 con 0x7fcad00716f0 2023-03-25T00:57:25.071 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad015cdd0 con 0x7fcad00716f0 2023-03-25T00:57:25.071 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcac5ffb700 1 -- 172.21.15.121:0/203105576 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcac00011a0 con 0x7fcad00716f0 2023-03-25T00:57:25.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcac5ffb700 1 -- 172.21.15.121:0/203105576 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcac0003680 con 0x7fcad00716f0 2023-03-25T00:57:25.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcac5ffb700 1 -- 172.21.15.121:0/203105576 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcac0003b80 con 0x7fcad00716f0 2023-03-25T00:57:25.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.064+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad00ce4f0 con 0x7fcad00716f0 2023-03-25T00:57:25.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.065+0000 7fcac5ffb700 1 -- 172.21.15.121:0/203105576 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fcac0011e00 con 0x7fcad00716f0 2023-03-25T00:57:25.072 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.065+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad00cc0b0 con 0x7fcad00716f0 2023-03-25T00:57:25.073 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.067+0000 7fcac5ffb700 1 -- 172.21.15.121:0/203105576 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fcac0017070 con 0x7fcad00716f0 2023-03-25T00:57:25.151 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.150+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "2c09166f-791d-42cf-9ef4-4746592f74e2", "id": 3} v 0) v1 -- 0x7fcad0071d40 con 0x7fcad00716f0 2023-03-25T00:57:25.158 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.157+0000 7fcac5ffb700 1 -- 172.21.15.121:0/203105576 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2c09166f-791d-42cf-9ef4-4746592f74e2", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcac0011590 con 0x7fcad00716f0 2023-03-25T00:57:25.159 INFO:teuthology.orchestra.run.smithi121.stdout:3 2023-03-25T00:57:25.160 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 msgr2=0x7fcad014dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.160 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad014dca0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcac000c450 tx=0x7fcac000c760 comp rx=0 tx=0).stop 2023-03-25T00:57:25.160 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 shutdown_connections 2023-03-25T00:57:25.160 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fcad015ca00 0x7fcad0156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.160 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fcad00716f0 0x7fcad014dca0 secure :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcac000c450 tx=0x7fcac000c760 comp rx=0 tx=0).stop 2023-03-25T00:57:25.160 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 --2- 172.21.15.121:0/203105576 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcad00b57c0 0x7fcad00c93f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 >> 172.21.15.121:0/203105576 conn(0x7fcad00d0b50 msgr2=0x7fcad00b4c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 shutdown_connections 2023-03-25T00:57:25.161 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.158+0000 7fcad733b700 1 -- 172.21.15.121:0/203105576 wait complete. 2023-03-25T00:57:25.171 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 2caea232-d588-4aac-bde0-39dae45860ac 4 2023-03-25T00:57:25.210 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:25.208+0000 7fa7398e6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-25T00:57:25.229 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:25.227+0000 7f9bea30b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-25T00:57:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.291+0000 7f933b778700 1 Processor -- start 2023-03-25T00:57:25.293 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.292+0000 7f933b778700 1 -- start start 2023-03-25T00:57:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.292+0000 7f933b778700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f9334159610 0x7f9334070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.292+0000 7f933b778700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f933415e030 0x7f93340cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.292+0000 7f933b778700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93340cc530 con 0x7f93341599e0 2023-03-25T00:57:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.292+0000 7f933b778700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93340cc250 con 0x7f9334159610 2023-03-25T00:57:25.295 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.292+0000 7f933b778700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f933415d550 con 0x7f933415e030 2023-03-25T00:57:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9339d15700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f933415e030 0x7f93340cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9338d13700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f93341599e0 0x7f93340cb520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54520/0 (socket says 172.21.15.121:54520) 2023-03-25T00:57:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9338d13700 1 -- 172.21.15.121:0/1879427857 learned_addr learned my addr 172.21.15.121:0/1879427857 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-25T00:57:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9339514700 1 --2- 172.21.15.121:0/1879427857 >> v2:172.21.15.115:3300/0 conn(0x7f9334159610 0x7f9334070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9339d15700 1 -- 172.21.15.121:0/1879427857 >> v2:172.21.15.115:3300/0 conn(0x7f9334159610 msgr2=0x7f9334070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9339d15700 1 --2- 172.21.15.121:0/1879427857 >> v2:172.21.15.115:3300/0 conn(0x7f9334159610 0x7f9334070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.296 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9339d15700 1 -- 172.21.15.121:0/1879427857 >> v1:172.21.15.121:6789/0 conn(0x7f93341599e0 legacy=0x7f93340cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9339d15700 1 -- 172.21.15.121:0/1879427857 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933415d830 con 0x7f933415e030 2023-03-25T00:57:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f932bfff700 1 -- 172.21.15.121:0/1879427857 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126905272 0 0) 0x7f93340cc530 con 0x7f93341599e0 2023-03-25T00:57:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f9339d15700 1 --2- 172.21.15.121:0/1879427857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933415e030 0x7f93340cbc30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9334072550 tx=0x7f932c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=365e5d59810c8370 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f932bfff700 1 -- 172.21.15.121:0/1879427857 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c00e070 con 0x7f933415e030 2023-03-25T00:57:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f932bfff700 1 -- 172.21.15.121:0/1879427857 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c00bb20 con 0x7f933415e030 2023-03-25T00:57:25.297 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.293+0000 7f932bfff700 1 -- 172.21.15.121:0/1879427857 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c012430 con 0x7f933415e030 2023-03-25T00:57:25.299 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.294+0000 7f933b778700 1 -- 172.21.15.121:0/1879427857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933415e030 msgr2=0x7f93340cbc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.294+0000 7f933b778700 1 --2- 172.21.15.121:0/1879427857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933415e030 0x7f93340cbc30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f9334072550 tx=0x7f932c00b330 comp rx=0 tx=0).stop 2023-03-25T00:57:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.294+0000 7f933b778700 1 -- 172.21.15.121:0/1879427857 shutdown_connections 2023-03-25T00:57:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.294+0000 7f933b778700 1 --2- 172.21.15.121:0/1879427857 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933415e030 0x7f93340cbc30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.294+0000 7f933b778700 1 --2- 172.21.15.121:0/1879427857 >> v2:172.21.15.115:3300/0 conn(0x7f9334159610 0x7f9334070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.294+0000 7f933b778700 1 -- 172.21.15.121:0/1879427857 >> 172.21.15.121:0/1879427857 conn(0x7f93340d0b50 msgr2=0x7f93340cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.294+0000 7f933b778700 1 -- 172.21.15.121:0/1879427857 shutdown_connections 2023-03-25T00:57:25.300 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.294+0000 7f933b778700 1 -- 172.21.15.121:0/1879427857 wait complete. 2023-03-25T00:57:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.295+0000 7f933b778700 1 Processor -- start 2023-03-25T00:57:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.295+0000 7f933b778700 1 -- start start 2023-03-25T00:57:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f933b778700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9334159610 0x7f93340c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f933b778700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93341599e0 0x7f93340b0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f933b778700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933415e030 0x7f9334147ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f933b778700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c4d00 con 0x7f933415e030 2023-03-25T00:57:25.301 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f933b778700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933415bdf0 con 0x7f9334159610 2023-03-25T00:57:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f933b778700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93340c3a00 con 0x7f93341599e0 2023-03-25T00:57:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f9339514700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9334159610 0x7f93340c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f9338d13700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93341599e0 0x7f93340b0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f9339514700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9334159610 0x7f93340c47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45008/0 (socket says 172.21.15.121:45008) 2023-03-25T00:57:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f9339514700 1 -- 172.21.15.121:0/4212699465 learned_addr learned my addr 172.21.15.121:0/4212699465 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f9338d13700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93341599e0 0x7f93340b0990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44892/0 (socket says 172.21.15.121:44892) 2023-03-25T00:57:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.296+0000 7f9339d15700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933415e030 0x7f9334147ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.302 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9339d15700 1 -- 172.21.15.121:0/4212699465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93341599e0 msgr2=0x7f93340b0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9339d15700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93341599e0 0x7f93340b0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9339d15700 1 -- 172.21.15.121:0/4212699465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9334159610 msgr2=0x7f93340c47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9339d15700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9334159610 0x7f93340c47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9339d15700 1 -- 172.21.15.121:0/4212699465 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c009a70 con 0x7f933415e030 2023-03-25T00:57:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9338d13700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93341599e0 0x7f93340b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9339514700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9334159610 0x7f93340c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:25.303 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9339d15700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933415e030 0x7f9334147ed0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f932c016000 tx=0x7f932c004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9329ffb700 1 -- 172.21.15.121:0/4212699465 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c012430 con 0x7f933415e030 2023-03-25T00:57:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9329ffb700 1 -- 172.21.15.121:0/4212699465 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f932c020070 con 0x7f933415e030 2023-03-25T00:57:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9334071520 con 0x7f933415e030 2023-03-25T00:57:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9329ffb700 1 -- 172.21.15.121:0/4212699465 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f932c004280 con 0x7f933415e030 2023-03-25T00:57:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933415db10 con 0x7f933415e030 2023-03-25T00:57:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9329ffb700 1 -- 172.21.15.121:0/4212699465 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f932c0046e0 con 0x7f933415e030 2023-03-25T00:57:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.297+0000 7f9329ffb700 1 -- 172.21.15.121:0/4212699465 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f932c022070 con 0x7f933415e030 2023-03-25T00:57:25.304 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.298+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93340716b0 con 0x7f933415e030 2023-03-25T00:57:25.305 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.300+0000 7f9329ffb700 1 -- 172.21.15.121:0/4212699465 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f932c045b10 con 0x7f933415e030 2023-03-25T00:57:25.386 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.385+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2caea232-d588-4aac-bde0-39dae45860ac", "id": 4} v 0) v1 -- 0x7f9334074580 con 0x7f933415e030 2023-03-25T00:57:25.393 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.392+0000 7f9329ffb700 1 -- 172.21.15.121:0/4212699465 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2caea232-d588-4aac-bde0-39dae45860ac", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f932c034030 con 0x7f933415e030 2023-03-25T00:57:25.394 INFO:teuthology.orchestra.run.smithi121.stdout:4 2023-03-25T00:57:25.395 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933415e030 msgr2=0x7f9334147ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.395 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933415e030 0x7f9334147ed0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f932c016000 tx=0x7f932c004ad0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.395 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 shutdown_connections 2023-03-25T00:57:25.395 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9334159610 0x7f93340c47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.395 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f933415e030 0x7f9334147ed0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.395 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 --2- 172.21.15.121:0/4212699465 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f93341599e0 0x7f93340b0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.396 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 >> 172.21.15.121:0/4212699465 conn(0x7f93340d0b50 msgr2=0x7f93340cf680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:25.396 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 shutdown_connections 2023-03-25T00:57:25.396 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.393+0000 7f933b778700 1 -- 172.21.15.121:0/4212699465 wait complete. 2023-03-25T00:57:25.406 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new cefc141f-68fb-4be4-957b-85ce0049ada5 5 2023-03-25T00:57:25.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.524+0000 7f7d92255700 1 Processor -- start 2023-03-25T00:57:25.526 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d92255700 1 -- start start 2023-03-25T00:57:25.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d92255700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f7d8c0cda00 0x7f7d8c0cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d92255700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d8c0cb960 0x7f7d8c0cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d92255700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8c0cc2e0 con 0x7f7d8c159940 2023-03-25T00:57:25.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d92255700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8c0cb170 con 0x7f7d8c0cda00 2023-03-25T00:57:25.528 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d92255700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8c0cb450 con 0x7f7d8c0cb960 2023-03-25T00:57:25.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d8bfff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f7d8c0cb960 0x7f7d8c0cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d8b7fe700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f7d8c159940 0x7f7d8c0b4ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54532/0 (socket says 172.21.15.121:54532) 2023-03-25T00:57:25.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.525+0000 7f7d8b7fe700 1 -- 172.21.15.121:0/2694767252 learned_addr learned my addr 172.21.15.121:0/2694767252 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-25T00:57:25.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8affd700 1 --2- 172.21.15.121:0/2694767252 >> v2:172.21.15.115:3300/0 conn(0x7f7d8c0cda00 0x7f7d8c0cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8bfff700 1 -- 172.21.15.121:0/2694767252 >> v2:172.21.15.115:3300/0 conn(0x7f7d8c0cda00 msgr2=0x7f7d8c0cde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:25.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8bfff700 1 --2- 172.21.15.121:0/2694767252 >> v2:172.21.15.115:3300/0 conn(0x7f7d8c0cda00 0x7f7d8c0cde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.529 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8a7fc700 1 -- 172.21.15.121:0/2694767252 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830558546 0 0) 0x7f7d8c0cc2e0 con 0x7f7d8c159940 2023-03-25T00:57:25.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8bfff700 1 -- 172.21.15.121:0/2694767252 >> v1:172.21.15.121:6789/0 conn(0x7f7d8c159940 legacy=0x7f7d8c0b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8bfff700 1 -- 172.21.15.121:0/2694767252 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8c0b42c0 con 0x7f7d8c0cb960 2023-03-25T00:57:25.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8bfff700 1 --2- 172.21.15.121:0/2694767252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0cbda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c067830 tx=0x7f7d8400b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e79224e0b93e02b server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:25.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8a7fc700 1 -- 172.21.15.121:0/2694767252 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d8400e070 con 0x7f7d8c0cb960 2023-03-25T00:57:25.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8a7fc700 1 -- 172.21.15.121:0/2694767252 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d8400bb20 con 0x7f7d8c0cb960 2023-03-25T00:57:25.530 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.526+0000 7f7d8a7fc700 1 -- 172.21.15.121:0/2694767252 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d84012430 con 0x7f7d8c0cb960 2023-03-25T00:57:25.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.527+0000 7f7d92255700 1 -- 172.21.15.121:0/2694767252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 msgr2=0x7f7d8c0cbda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.527+0000 7f7d92255700 1 --2- 172.21.15.121:0/2694767252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0cbda0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7d8c067830 tx=0x7f7d8400b330 comp rx=0 tx=0).stop 2023-03-25T00:57:25.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.527+0000 7f7d92255700 1 -- 172.21.15.121:0/2694767252 shutdown_connections 2023-03-25T00:57:25.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.527+0000 7f7d92255700 1 --2- 172.21.15.121:0/2694767252 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0cbda0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.527+0000 7f7d92255700 1 --2- 172.21.15.121:0/2694767252 >> v2:172.21.15.115:3300/0 conn(0x7f7d8c0cda00 0x7f7d8c0cde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.527+0000 7f7d92255700 1 -- 172.21.15.121:0/2694767252 >> 172.21.15.121:0/2694767252 conn(0x7f7d8c0d0b50 msgr2=0x7f7d8c0cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:25.533 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.527+0000 7f7d92255700 1 -- 172.21.15.121:0/2694767252 shutdown_connections 2023-03-25T00:57:25.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.527+0000 7f7d92255700 1 -- 172.21.15.121:0/2694767252 wait complete. 2023-03-25T00:57:25.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.528+0000 7f7d92255700 1 Processor -- start 2023-03-25T00:57:25.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.529+0000 7f7d92255700 1 -- start start 2023-03-25T00:57:25.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d92255700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d92255700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d8c0cda00 0x7f7d8c14a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d92255700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d8c159940 0x7f7d8c153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.534 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d92255700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c0c7b60 con 0x7f7d8c159940 2023-03-25T00:57:25.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d92255700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c0c7e40 con 0x7f7d8c0cb960 2023-03-25T00:57:25.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d92255700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c0c8120 con 0x7f7d8c0cda00 2023-03-25T00:57:25.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45020/0 (socket says 172.21.15.121:45020) 2023-03-25T00:57:25.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 -- 172.21.15.121:0/808368157 learned_addr learned my addr 172.21.15.121:0/808368157 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:25.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8affd700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d8c0cda00 0x7f7d8c14a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8bfff700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d8c159940 0x7f7d8c153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.535 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 -- 172.21.15.121:0/808368157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d8c0cda00 msgr2=0x7f7d8c14a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d8c0cda00 0x7f7d8c14a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 -- 172.21.15.121:0/808368157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d8c159940 msgr2=0x7f7d8c153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d8c159940 0x7f7d8c153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 -- 172.21.15.121:0/808368157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d84009a70 con 0x7f7d8c0cb960 2023-03-25T00:57:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d8b7fe700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0c6f00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c0011e0 tx=0x7f7d7c00ca30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d73fff700 1 -- 172.21.15.121:0/808368157 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c004c80 con 0x7f7d8c0cb960 2023-03-25T00:57:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d73fff700 1 -- 172.21.15.121:0/808368157 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d7c004e00 con 0x7f7d8c0cb960 2023-03-25T00:57:25.536 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d73fff700 1 -- 172.21.15.121:0/808368157 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d7c007310 con 0x7f7d8c0cb960 2023-03-25T00:57:25.537 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8c0b51d0 con 0x7f7d8c0cb960 2023-03-25T00:57:25.537 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.530+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8c0b45a0 con 0x7f7d8c0cb960 2023-03-25T00:57:25.537 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.531+0000 7f7d73fff700 1 -- 172.21.15.121:0/808368157 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f7d7c011470 con 0x7f7d8c0cb960 2023-03-25T00:57:25.537 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.531+0000 7f7d73fff700 1 -- 172.21.15.121:0/808368157 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f7d7c011ca0 con 0x7f7d8c0cb960 2023-03-25T00:57:25.537 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.531+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8c0b3f00 con 0x7f7d8c0cb960 2023-03-25T00:57:25.538 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.533+0000 7f7d73fff700 1 -- 172.21.15.121:0/808368157 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f7d7c011720 con 0x7f7d8c0cb960 2023-03-25T00:57:25.623 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.621+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cefc141f-68fb-4be4-957b-85ce0049ada5", "id": 5} v 0) v1 -- 0x7f7d8c074580 con 0x7f7d8c0cb960 2023-03-25T00:57:25.630 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.629+0000 7f7d73fff700 1 -- 172.21.15.121:0/808368157 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cefc141f-68fb-4be4-957b-85ce0049ada5", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f7d7c0118c0 con 0x7f7d8c0cb960 2023-03-25T00:57:25.631 INFO:teuthology.orchestra.run.smithi121.stdout:5 2023-03-25T00:57:25.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 msgr2=0x7f7d8c0c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0c6f00 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7d7c0011e0 tx=0x7f7d7c00ca30 comp rx=0 tx=0).stop 2023-03-25T00:57:25.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 shutdown_connections 2023-03-25T00:57:25.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7d8c0cb960 0x7f7d8c0c6f00 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.632 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f7d8c159940 0x7f7d8c153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 --2- 172.21.15.121:0/808368157 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f7d8c0cda00 0x7f7d8c14a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 >> 172.21.15.121:0/808368157 conn(0x7f7d8c0d0b50 msgr2=0x7f7d8c157820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:25.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 shutdown_connections 2023-03-25T00:57:25.633 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.630+0000 7f7d92255700 1 -- 172.21.15.121:0/808368157 wait complete. 2023-03-25T00:57:25.646 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 325c5ead-d0cd-45e2-b16c-d967f661d738 6 2023-03-25T00:57:25.776 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.775+0000 7f01e95d8700 1 Processor -- start 2023-03-25T00:57:25.777 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.775+0000 7f01e95d8700 1 -- start start 2023-03-25T00:57:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e95d8700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f01e415caf0 0x7f01e40ad740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e95d8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40c9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e95d8700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01e415cec0 con 0x7f01e40cde60 2023-03-25T00:57:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e95d8700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01e40add20 con 0x7f01e415caf0 2023-03-25T00:57:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e95d8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e40ae000 con 0x7f01e4070f70 2023-03-25T00:57:25.779 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e2ffd700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f01e415caf0 0x7f01e40ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e37fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40c9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e27fc700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f01e40cde60 0x7f01e4070860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.121:54544/0 (socket says 172.21.15.121:54544) 2023-03-25T00:57:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e27fc700 1 -- 172.21.15.121:0/3117368764 learned_addr learned my addr 172.21.15.121:0/3117368764 (peer_addr_for_me v1:172.21.15.121:0/0) 2023-03-25T00:57:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e37fe700 1 -- 172.21.15.121:0/3117368764 >> v2:172.21.15.121:3300/0 conn(0x7f01e415caf0 msgr2=0x7f01e40ad740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e37fe700 1 --2- 172.21.15.121:0/3117368764 >> v2:172.21.15.121:3300/0 conn(0x7f01e415caf0 0x7f01e40ad740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e37fe700 1 -- 172.21.15.121:0/3117368764 >> v1:172.21.15.121:6789/0 conn(0x7f01e40cde60 legacy=0x7f01e4070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.780 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e37fe700 1 -- 172.21.15.121:0/3117368764 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e40ae2e0 con 0x7f01e4070f70 2023-03-25T00:57:25.781 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e2ffd700 1 --2- 172.21.15.121:0/3117368764 >> v2:172.21.15.121:3300/0 conn(0x7f01e415caf0 0x7f01e40ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:25.781 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e37fe700 1 --2- 172.21.15.121:0/3117368764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40c9c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009dc0 tx=0x7f01dc00b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=41e1a33d81c2f274 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:25.781 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e1ffb700 1 -- 172.21.15.121:0/3117368764 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc00e070 con 0x7f01e4070f70 2023-03-25T00:57:25.781 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e1ffb700 1 -- 172.21.15.121:0/3117368764 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01dc00bae0 con 0x7f01e4070f70 2023-03-25T00:57:25.781 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.776+0000 7f01e1ffb700 1 -- 172.21.15.121:0/3117368764 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01dc00be90 con 0x7f01e4070f70 2023-03-25T00:57:25.781 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.777+0000 7f01e95d8700 1 -- 172.21.15.121:0/3117368764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 msgr2=0x7f01e40c9c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.781 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.777+0000 7f01e95d8700 1 --2- 172.21.15.121:0/3117368764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40c9c80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f01dc009dc0 tx=0x7f01dc00b330 comp rx=0 tx=0).stop 2023-03-25T00:57:25.782 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.777+0000 7f01e95d8700 1 -- 172.21.15.121:0/3117368764 shutdown_connections 2023-03-25T00:57:25.782 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.777+0000 7f01e95d8700 1 --2- 172.21.15.121:0/3117368764 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40c9c80 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.782 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.777+0000 7f01e95d8700 1 --2- 172.21.15.121:0/3117368764 >> v2:172.21.15.121:3300/0 conn(0x7f01e415caf0 0x7f01e40ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.782 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.777+0000 7f01e95d8700 1 -- 172.21.15.121:0/3117368764 >> 172.21.15.121:0/3117368764 conn(0x7f01e40ccd40 msgr2=0x7f01e40cf630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:25.782 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.777+0000 7f01e95d8700 1 -- 172.21.15.121:0/3117368764 shutdown_connections 2023-03-25T00:57:25.782 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.777+0000 7f01e95d8700 1 -- 172.21.15.121:0/3117368764 wait complete. 2023-03-25T00:57:25.784 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.779+0000 7f01e95d8700 1 Processor -- start 2023-03-25T00:57:25.784 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.779+0000 7f01e95d8700 1 -- start start 2023-03-25T00:57:25.785 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e95d8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.785 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e95d8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e40cde60 0x7f01e40b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.785 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e95d8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01e415caf0 0x7f01e4145d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:25.785 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e95d8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e40b3d00 con 0x7f01e415caf0 2023-03-25T00:57:25.785 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e95d8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e40b3fe0 con 0x7f01e40cde60 2023-03-25T00:57:25.785 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e95d8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01e40b42c0 con 0x7f01e4070f70 2023-03-25T00:57:25.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.121:44914/0 (socket says 172.21.15.121:44914) 2023-03-25T00:57:25.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 -- 172.21.15.121:0/2171737235 learned_addr learned my addr 172.21.15.121:0/2171737235 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:25.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e27fc700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e40cde60 0x7f01e40b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:25.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 -- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e40cde60 msgr2=0x7f01e40b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e40cde60 0x7f01e40b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.786 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 -- 172.21.15.121:0/2171737235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01e415caf0 msgr2=0x7f01e4145d70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-03-25T00:57:25.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01e415caf0 0x7f01e4145d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 -- 172.21.15.121:0/2171737235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d40094a0 con 0x7f01e4070f70 2023-03-25T00:57:25.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e27fc700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e40cde60 0x7f01e40b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:25.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01e2ffd700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40b34d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f01d400ebe0 tx=0x7f01d400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:25.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.780+0000 7f01cb7fe700 1 -- 172.21.15.121:0/2171737235 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d400c850 con 0x7f01e4070f70 2023-03-25T00:57:25.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.781+0000 7f01cb7fe700 1 -- 172.21.15.121:0/2171737235 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01d400c9b0 con 0x7f01e4070f70 2023-03-25T00:57:25.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.781+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01dc009a70 con 0x7f01e4070f70 2023-03-25T00:57:25.787 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.781+0000 7f01cb7fe700 1 -- 172.21.15.121:0/2171737235 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01d400cd20 con 0x7f01e4070f70 2023-03-25T00:57:25.788 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.781+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e40ae8a0 con 0x7f01e4070f70 2023-03-25T00:57:25.788 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.781+0000 7f01cb7fe700 1 -- 172.21.15.121:0/2171737235 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f01d4005d20 con 0x7f01e4070f70 2023-03-25T00:57:25.788 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.781+0000 7f01cb7fe700 1 -- 172.21.15.121:0/2171737235 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f01d4006890 con 0x7f01e4070f70 2023-03-25T00:57:25.788 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.782+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e40ae5c0 con 0x7f01e4070f70 2023-03-25T00:57:25.789 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:25.785+0000 7fa7398e6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-25T00:57:25.789 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.784+0000 7f01cb7fe700 1 -- 172.21.15.121:0/2171737235 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f01d4021060 con 0x7f01e4070f70 2023-03-25T00:57:25.812 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:25.810+0000 7f9bea30b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-25T00:57:25.868 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.867+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd new", "uuid": "325c5ead-d0cd-45e2-b16c-d967f661d738", "id": 6} v 0) v1 -- 0x7f01e40aedc0 con 0x7f01e4070f70 2023-03-25T00:57:25.876 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.875+0000 7f01cb7fe700 1 -- 172.21.15.121:0/2171737235 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "325c5ead-d0cd-45e2-b16c-d967f661d738", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f01d402a0b0 con 0x7f01e4070f70 2023-03-25T00:57:25.877 INFO:teuthology.orchestra.run.smithi121.stdout:6 2023-03-25T00:57:25.877 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 msgr2=0x7f01e40b34d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40b34d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f01d400ebe0 tx=0x7f01d400eef0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 shutdown_connections 2023-03-25T00:57:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f01e415caf0 0x7f01e4145d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f01e4070f70 0x7f01e40b34d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 --2- 172.21.15.121:0/2171737235 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f01e40cde60 0x7f01e40b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 >> 172.21.15.121:0/2171737235 conn(0x7f01e40ccd40 msgr2=0x7f01e415bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:25.878 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 shutdown_connections 2023-03-25T00:57:25.879 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:25.876+0000 7f01e95d8700 1 -- 172.21.15.121:0/2171737235 wait complete. 2023-03-25T00:57:25.890 DEBUG:teuthology.orchestra.run.smithi121:> sudo ceph --cluster ceph osd new 274ccf94-d180-4a8c-a0c3-f620f738252c 7 2023-03-25T00:57:25.941 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:25.939+0000 7fa7398e6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-25T00:57:25.972 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:25.970+0000 7f9bea30b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-25T00:57:26.010 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.009+0000 7f6b47d04700 1 Processor -- start 2023-03-25T00:57:26.011 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.009+0000 7f6b47d04700 1 -- start start 2023-03-25T00:57:26.016 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b47d04700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6b40159610 0x7f6b40070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b47d04700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6b4015e030 0x7f6b400cbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b47d04700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b400cc530 con 0x7f6b401599e0 2023-03-25T00:57:26.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b47d04700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b400cc250 con 0x7f6b4015e030 2023-03-25T00:57:26.017 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b47d04700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b4015d550 con 0x7f6b40159610 2023-03-25T00:57:26.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b45aa0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6b40159610 0x7f6b40070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b45aa0700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f6b40159610 0x7f6b40070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45038/0 (socket says 172.21.15.121:45038) 2023-03-25T00:57:26.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b45aa0700 1 -- 172.21.15.121:0/3466852774 learned_addr learned my addr 172.21.15.121:0/3466852774 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:26.018 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b462a1700 1 --2- 172.21.15.121:0/3466852774 >> v2:172.21.15.115:3300/0 conn(0x7f6b4015e030 0x7f6b400cbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b45aa0700 1 -- 172.21.15.121:0/3466852774 >> v2:172.21.15.115:3300/0 conn(0x7f6b4015e030 msgr2=0x7f6b400cbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:26.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b45aa0700 1 --2- 172.21.15.121:0/3466852774 >> v2:172.21.15.115:3300/0 conn(0x7f6b4015e030 0x7f6b400cbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.019 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b45aa0700 1 -- 172.21.15.121:0/3466852774 >> v1:172.21.15.115:6789/0 conn(0x7f6b401599e0 legacy=0x7f6b400cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b45aa0700 1 -- 172.21.15.121:0/3466852774 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4015d830 con 0x7f6b40159610 2023-03-25T00:57:26.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b45aa0700 1 --2- 172.21.15.121:0/3466852774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b40159610 0x7f6b40070fe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6b40067730 tx=0x7f6b3c00be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66c8cd91773e923c server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:26.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b44a9e700 1 -- 172.21.15.121:0/3466852774 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c00dc70 con 0x7f6b40159610 2023-03-25T00:57:26.020 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b44a9e700 1 -- 172.21.15.121:0/3466852774 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c00ddd0 con 0x7f6b40159610 2023-03-25T00:57:26.021 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.010+0000 7f6b44a9e700 1 -- 172.21.15.121:0/3466852774 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c010690 con 0x7f6b40159610 2023-03-25T00:57:26.021 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.011+0000 7f6b47d04700 1 -- 172.21.15.121:0/3466852774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b40159610 msgr2=0x7f6b40070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.021 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.011+0000 7f6b47d04700 1 --2- 172.21.15.121:0/3466852774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b40159610 0x7f6b40070fe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6b40067730 tx=0x7f6b3c00be70 comp rx=0 tx=0).stop 2023-03-25T00:57:26.021 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.011+0000 7f6b47d04700 1 -- 172.21.15.121:0/3466852774 shutdown_connections 2023-03-25T00:57:26.022 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.011+0000 7f6b47d04700 1 --2- 172.21.15.121:0/3466852774 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b40159610 0x7f6b40070fe0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.022 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.011+0000 7f6b47d04700 1 --2- 172.21.15.121:0/3466852774 >> v2:172.21.15.115:3300/0 conn(0x7f6b4015e030 0x7f6b400cbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.022 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.011+0000 7f6b47d04700 1 -- 172.21.15.121:0/3466852774 >> 172.21.15.121:0/3466852774 conn(0x7f6b400d0b50 msgr2=0x7f6b400cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:26.022 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.011+0000 7f6b47d04700 1 -- 172.21.15.121:0/3466852774 shutdown_connections 2023-03-25T00:57:26.023 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.011+0000 7f6b47d04700 1 -- 172.21.15.121:0/3466852774 wait complete. 2023-03-25T00:57:26.023 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.012+0000 7f6b47d04700 1 Processor -- start 2023-03-25T00:57:26.023 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.012+0000 7f6b47d04700 1 -- start start 2023-03-25T00:57:26.023 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b47d04700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b40159610 0x7f6b40152820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.023 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b47d04700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b401599e0 0x7f6b401501e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.023 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b47d04700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b4015e030 0x7f6b40150720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b47d04700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b40153050 con 0x7f6b40159610 2023-03-25T00:57:26.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b47d04700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b400c57e0 con 0x7f6b401599e0 2023-03-25T00:57:26.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b47d04700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b400c3b00 con 0x7f6b4015e030 2023-03-25T00:57:26.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b40159610 0x7f6b40152820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b40159610 0x7f6b40152820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.121:59678/0 (socket says 172.21.15.121:59678) 2023-03-25T00:57:26.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b462a1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b4015e030 0x7f6b40150720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 -- 172.21.15.121:0/3795425620 learned_addr learned my addr 172.21.15.121:0/3795425620 (peer_addr_for_me v2:172.21.15.121:0/0) 2023-03-25T00:57:26.024 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b4529f700 1 --2- 172.21.15.121:0/3795425620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b401599e0 0x7f6b401501e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.025 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 -- 172.21.15.121:0/3795425620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b4015e030 msgr2=0x7f6b40150720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.025 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 --2- 172.21.15.121:0/3795425620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b4015e030 0x7f6b40150720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.025 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 -- 172.21.15.121:0/3795425620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b401599e0 msgr2=0x7f6b401501e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.025 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 --2- 172.21.15.121:0/3795425620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b401599e0 0x7f6b401501e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.025 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 -- 172.21.15.121:0/3795425620 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b3c00b7f0 con 0x7f6b40159610 2023-03-25T00:57:26.025 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.013+0000 7f6b45aa0700 1 --2- 172.21.15.121:0/3795425620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b40159610 0x7f6b40152820 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00beb0 tx=0x7f6b3c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.014+0000 7f6b367fc700 1 -- 172.21.15.121:0/3795425620 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c01f070 con 0x7f6b40159610 2023-03-25T00:57:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.014+0000 7f6b367fc700 1 -- 172.21.15.121:0/3795425620 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b3c0038b0 con 0x7f6b40159610 2023-03-25T00:57:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.014+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b40071520 con 0x7f6b40159610 2023-03-25T00:57:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.014+0000 7f6b367fc700 1 -- 172.21.15.121:0/3795425620 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b3c003c80 con 0x7f6b40159610 2023-03-25T00:57:26.026 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.014+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b4015db10 con 0x7f6b40159610 2023-03-25T00:57:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.014+0000 7f6b367fc700 1 -- 172.21.15.121:0/3795425620 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6b3c010770 con 0x7f6b40159610 2023-03-25T00:57:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.014+0000 7f6b367fc700 1 -- 172.21.15.121:0/3795425620 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f6b3c003de0 con 0x7f6b40159610 2023-03-25T00:57:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.015+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b400716b0 con 0x7f6b40159610 2023-03-25T00:57:26.027 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.017+0000 7f6b367fc700 1 -- 172.21.15.121:0/3795425620 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f6b3c028070 con 0x7f6b40159610 2023-03-25T00:57:26.101 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.099+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd new", "uuid": "274ccf94-d180-4a8c-a0c3-f620f738252c", "id": 7} v 0) v1 -- 0x7f6b400c5640 con 0x7f6b40159610 2023-03-25T00:57:26.108 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.106+0000 7f6b367fc700 1 -- 172.21.15.121:0/3795425620 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "274ccf94-d180-4a8c-a0c3-f620f738252c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f6b400c5640 con 0x7f6b40159610 2023-03-25T00:57:26.108 INFO:teuthology.orchestra.run.smithi121.stdout:7 2023-03-25T00:57:26.109 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b40159610 msgr2=0x7f6b40152820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 --2- 172.21.15.121:0/3795425620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b40159610 0x7f6b40152820 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c00beb0 tx=0x7f6b3c004750 comp rx=0 tx=0).stop 2023-03-25T00:57:26.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 shutdown_connections 2023-03-25T00:57:26.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 --2- 172.21.15.121:0/3795425620 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6b4015e030 0x7f6b40150720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 --2- 172.21.15.121:0/3795425620 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6b40159610 0x7f6b40152820 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 --2- 172.21.15.121:0/3795425620 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6b401599e0 0x7f6b401501e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.110 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 >> 172.21.15.121:0/3795425620 conn(0x7f6b400d0b50 msgr2=0x7f6b400ced70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 shutdown_connections 2023-03-25T00:57:26.111 INFO:teuthology.orchestra.run.smithi121.stderr:2023-03-25T00:57:26.107+0000 7f6b47d04700 1 -- 172.21.15.121:0/3795425620 wait complete. 2023-03-25T00:57:26.120 INFO:tasks.ceph.osd.0:Restarting daemon 2023-03-25T00:57:26.120 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-03-25T00:57:26.122 INFO:tasks.ceph.osd.0:Started 2023-03-25T00:57:26.123 INFO:tasks.ceph.osd.1:Restarting daemon 2023-03-25T00:57:26.123 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-03-25T00:57:26.125 INFO:tasks.ceph.osd.1:Started 2023-03-25T00:57:26.125 INFO:tasks.ceph.osd.2:Restarting daemon 2023-03-25T00:57:26.126 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-03-25T00:57:26.128 INFO:tasks.ceph.osd.2:Started 2023-03-25T00:57:26.128 INFO:tasks.ceph.osd.3:Restarting daemon 2023-03-25T00:57:26.128 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-03-25T00:57:26.130 INFO:tasks.ceph.osd.3:Started 2023-03-25T00:57:26.130 INFO:tasks.ceph.osd.4:Restarting daemon 2023-03-25T00:57:26.131 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-03-25T00:57:26.132 INFO:tasks.ceph.osd.4:Started 2023-03-25T00:57:26.133 INFO:tasks.ceph.osd.5:Restarting daemon 2023-03-25T00:57:26.133 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-03-25T00:57:26.135 INFO:tasks.ceph.osd.5:Started 2023-03-25T00:57:26.135 INFO:tasks.ceph.osd.6:Restarting daemon 2023-03-25T00:57:26.135 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-03-25T00:57:26.137 INFO:tasks.ceph.osd.6:Started 2023-03-25T00:57:26.137 INFO:tasks.ceph.osd.7:Restarting daemon 2023-03-25T00:57:26.137 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-03-25T00:57:26.140 INFO:tasks.ceph.osd.7:Started 2023-03-25T00:57:26.140 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-25T00:57:26.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.287+0000 7f6aa5512700 1 Processor -- start 2023-03-25T00:57:26.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.287+0000 7f6aa5512700 1 -- start start 2023-03-25T00:57:26.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6aa5512700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 0x7f6aa00cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.291 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6aa5512700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6aa00b1800 0x7f6aa00cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6aa5512700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa00cbbe0 con 0x7f6aa0161270 2023-03-25T00:57:26.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6aa5512700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa00cbec0 con 0x7f6aa00b1800 2023-03-25T00:57:26.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6aa5512700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa0160790 con 0x7f6aa015c8b0 2023-03-25T00:57:26.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9f7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6aa00b1800 0x7f6aa00cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9f7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f6aa00b1800 0x7f6aa00cb5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60694/0 (socket says 172.21.15.115:60694) 2023-03-25T00:57:26.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9f7fe700 1 -- 172.21.15.115:0/1283570516 learned_addr learned my addr 172.21.15.115:0/1283570516 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:26.292 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9effd700 1 --2- 172.21.15.115:0/1283570516 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 0x7f6aa00cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9f7fe700 1 -- 172.21.15.115:0/1283570516 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 msgr2=0x7f6aa00cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:26.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9f7fe700 1 --2- 172.21.15.115:0/1283570516 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 0x7f6aa00cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9dffb700 1 -- 172.21.15.115:0/1283570516 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3726106628 0 0) 0x7f6aa00cbbe0 con 0x7f6aa0161270 2023-03-25T00:57:26.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9f7fe700 1 -- 172.21.15.115:0/1283570516 >> v1:172.21.15.121:6789/0 conn(0x7f6aa0161270 legacy=0x7f6aa00b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9f7fe700 1 -- 172.21.15.115:0/1283570516 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa0160a70 con 0x7f6aa00b1800 2023-03-25T00:57:26.293 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.288+0000 7f6a9f7fe700 1 --2- 172.21.15.115:0/1283570516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa00b1800 0x7f6aa00cb5d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009e50 tx=0x7f6a9800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47e322bc02ad1a5 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:26.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6a9dffb700 1 -- 172.21.15.115:0/1283570516 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9800e070 con 0x7f6aa00b1800 2023-03-25T00:57:26.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6a9dffb700 1 -- 172.21.15.115:0/1283570516 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a9800b910 con 0x7f6aa00b1800 2023-03-25T00:57:26.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6a9dffb700 1 -- 172.21.15.115:0/1283570516 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9800bcc0 con 0x7f6aa00b1800 2023-03-25T00:57:26.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6aa5512700 1 -- 172.21.15.115:0/1283570516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa00b1800 msgr2=0x7f6aa00cb5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6aa5512700 1 --2- 172.21.15.115:0/1283570516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa00b1800 0x7f6aa00cb5d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f6a98009e50 tx=0x7f6a9800b330 comp rx=0 tx=0).stop 2023-03-25T00:57:26.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6aa5512700 1 -- 172.21.15.115:0/1283570516 shutdown_connections 2023-03-25T00:57:26.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6aa5512700 1 --2- 172.21.15.115:0/1283570516 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa00b1800 0x7f6aa00cb5d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.294 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6aa5512700 1 --2- 172.21.15.115:0/1283570516 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 0x7f6aa00cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6aa5512700 1 -- 172.21.15.115:0/1283570516 >> 172.21.15.115:0/1283570516 conn(0x7f6aa006d880 msgr2=0x7f6aa00cc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:26.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6aa5512700 1 -- 172.21.15.115:0/1283570516 shutdown_connections 2023-03-25T00:57:26.295 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.289+0000 7f6aa5512700 1 -- 172.21.15.115:0/1283570516 wait complete. 2023-03-25T00:57:26.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6aa5512700 1 Processor -- start 2023-03-25T00:57:26.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6aa5512700 1 -- start start 2023-03-25T00:57:26.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6aa5512700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa00b1800 0x7f6aa015d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6aa5512700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 0x7f6aa015d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6aa5512700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa0161270 0x7f6aa015e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:26.297 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6aa5512700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa015e140 con 0x7f6aa0161270 2023-03-25T00:57:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6aa5512700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa01f1df0 con 0x7f6aa00b1800 2023-03-25T00:57:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6aa5512700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa0155020 con 0x7f6aa015c8b0 2023-03-25T00:57:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6a9f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa0161270 0x7f6aa015e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6a9f7fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa0161270 0x7f6aa015e610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60698/0 (socket says 172.21.15.115:60698) 2023-03-25T00:57:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.291+0000 7f6a9f7fe700 1 -- 172.21.15.115:0/504080652 learned_addr learned my addr 172.21.15.115:0/504080652 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a9effd700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa00b1800 0x7f6aa015d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a9e7fc700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 0x7f6aa015d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:26.298 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a9f7fe700 1 -- 172.21.15.115:0/504080652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 msgr2=0x7f6aa015d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a9f7fe700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 0x7f6aa015d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a9f7fe700 1 -- 172.21.15.115:0/504080652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa00b1800 msgr2=0x7f6aa015d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a9f7fe700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa00b1800 0x7f6aa015d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a9f7fe700 1 -- 172.21.15.115:0/504080652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a98009b30 con 0x7f6aa0161270 2023-03-25T00:57:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a9f7fe700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa0161270 0x7f6aa015e610 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0073730 tx=0x7f6a98004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a877fe700 1 -- 172.21.15.115:0/504080652 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a98004880 con 0x7f6aa0161270 2023-03-25T00:57:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a877fe700 1 -- 172.21.15.115:0/504080652 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a980099b0 con 0x7f6aa0161270 2023-03-25T00:57:26.299 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa0072a50 con 0x7f6aa0161270 2023-03-25T00:57:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6a877fe700 1 -- 172.21.15.115:0/504080652 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a9801c680 con 0x7f6aa0161270 2023-03-25T00:57:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.292+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa0160d50 con 0x7f6aa0161270 2023-03-25T00:57:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.293+0000 7f6a877fe700 1 -- 172.21.15.115:0/504080652 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f6a9801cae0 con 0x7f6aa0161270 2023-03-25T00:57:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.293+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa015cc80 con 0x7f6aa0161270 2023-03-25T00:57:26.300 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.293+0000 7f6a877fe700 1 -- 172.21.15.115:0/504080652 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f6a98012ac0 con 0x7f6aa0161270 2023-03-25T00:57:26.301 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.295+0000 7f6a877fe700 1 -- 172.21.15.115:0/504080652 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f6a98020070 con 0x7f6aa0161270 2023-03-25T00:57:26.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.364+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6aa0074580 con 0x7f6aa0161270 2023-03-25T00:57:26.366 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.364+0000 7f6a877fe700 1 -- 172.21.15.115:0/504080652 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f6a98033020 con 0x7f6aa0161270 2023-03-25T00:57:26.367 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:57:26.367 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":10,"fsid":"9c1ea6b6-7c28-4560-af27-3b583d106388","created":"2023-03-25T00:57:23.615145+0000","modified":"2023-03-25T00:57:26.100484+0000","last_up_change":"0.000000","last_in_change":"2023-03-25T00:57:26.100484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6e324a3-c8ad-459c-bb1e-256d32a317db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"009b7c98-0c48-49f2-98a1-e8d7afe9af58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b4056d77-ca36-481f-9779-017fb707bda5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c09166f-791d-42cf-9ef4-4746592f74e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2caea232-d588-4aac-bde0-39dae45860ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cefc141f-68fb-4be4-957b-85ce0049ada5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"325c5ead-d0cd-45e2-b16c-d967f661d738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"274ccf94-d180-4a8c-a0c3-f620f738252c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-25T00:57:26.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa0161270 msgr2=0x7f6aa015e610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:26.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa0161270 0x7f6aa015e610 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6aa0073730 tx=0x7f6a98004ad0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 shutdown_connections 2023-03-25T00:57:26.368 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6aa00b1800 0x7f6aa015d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f6aa0161270 0x7f6aa015e610 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 --2- 172.21.15.115:0/504080652 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f6aa015c8b0 0x7f6aa015d550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:26.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 >> 172.21.15.115:0/504080652 conn(0x7f6aa006d880 msgr2=0x7f6aa00ac8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:26.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 shutdown_connections 2023-03-25T00:57:26.369 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:26.365+0000 7f6aa5512700 1 -- 172.21.15.115:0/504080652 wait complete. 2023-03-25T00:57:26.379 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-03-25T00:57:26.379 INFO:tasks.ceph:Waiting for OSDs to come up 2023-03-25T00:57:26.499 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-03-25T00:57:26.496+0000 7fdcd06e5640 -1 Falling back to public interface 2023-03-25T00:57:26.678 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:26.676+0000 7fa7398e6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-25T00:57:26.679 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:26.678+0000 7f9bea30b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-25T00:57:26.738 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-03-25T00:57:26.736+0000 7fdebf5cf640 -1 Falling back to public interface 2023-03-25T00:57:26.742 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-03-25T00:57:26.740+0000 7ff4c11c4640 -1 Falling back to public interface 2023-03-25T00:57:26.749 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-03-25T00:57:26.748+0000 7f11fc7d3640 -1 Falling back to public interface 2023-03-25T00:57:26.752 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-03-25T00:57:26.750+0000 7f7f65345640 -1 Falling back to public interface 2023-03-25T00:57:26.755 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:26.754+0000 7fa7398e6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-25T00:57:26.757 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:26.756+0000 7f9bea30b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-25T00:57:26.758 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-03-25T00:57:26.755+0000 7fce256d6640 -1 Falling back to public interface 2023-03-25T00:57:26.763 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-03-25T00:57:26.761+0000 7f71331cd640 -1 Falling back to public interface 2023-03-25T00:57:26.770 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-03-25T00:57:26.768+0000 7f832ae48640 -1 Falling back to public interface 2023-03-25T00:57:26.897 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:26.896+0000 7f9bea30b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-25T00:57:26.898 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:26.896+0000 7fa7398e6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-25T00:57:27.128 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:27.126+0000 7f9bea30b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-25T00:57:27.131 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:27.129+0000 7fa7398e6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-25T00:57:27.184 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-25T00:57:27.200 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:27.199+0000 7f9bea30b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-25T00:57:27.204 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:27.202+0000 7fa7398e6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-25T00:57:27.298 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.296+0000 7f2fe11b5700 1 Processor -- start 2023-03-25T00:57:27.298 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.296+0000 7f2fe11b5700 1 -- start start 2023-03-25T00:57:27.300 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fe11b5700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f2fdc15dd30 0x7f2fdc0cb880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:27.300 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fe11b5700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fdc0cc200 con 0x7f2fdc071010 2023-03-25T00:57:27.300 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fe11b5700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fdc0b4200 con 0x7f2fdc15c900 2023-03-25T00:57:27.300 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fe11b5700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fdc0b44e0 con 0x7f2fdc15dd30 2023-03-25T00:57:27.300 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fdb59e700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f2fdc071010 0x7f2fdc15e120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33034/0 (socket says 172.21.15.115:33034) 2023-03-25T00:57:27.301 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fdb59e700 1 -- 172.21.15.115:0/2937298011 learned_addr learned my addr 172.21.15.115:0/2937298011 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:27.301 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fda59c700 1 --2- 172.21.15.115:0/2937298011 >> v2:172.21.15.121:3300/0 conn(0x7f2fdc15dd30 0x7f2fdc0cb880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:27.301 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fd9d9b700 1 -- 172.21.15.115:0/2937298011 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943158909 0 0) 0x7f2fdc0cc200 con 0x7f2fdc071010 2023-03-25T00:57:27.301 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fd9d9b700 1 -- 172.21.15.115:0/2937298011 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fc40035b0 con 0x7f2fdc071010 2023-03-25T00:57:27.301 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fda59c700 1 -- 172.21.15.115:0/2937298011 >> v1:172.21.15.121:6789/0 conn(0x7f2fdc15c900 legacy=0x7f2fdc0b4ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:27.301 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fda59c700 1 -- 172.21.15.115:0/2937298011 >> v1:172.21.15.115:6789/0 conn(0x7f2fdc071010 legacy=0x7f2fdc15e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:27.301 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.297+0000 7f2fd9d9b700 1 -- 172.21.15.115:0/2937298011 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260704648 0 0) 0x7f2fdc0b4200 con 0x7f2fdc15c900 2023-03-25T00:57:27.302 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.298+0000 7f2fda59c700 1 -- 172.21.15.115:0/2937298011 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc0b47c0 con 0x7f2fdc15dd30 2023-03-25T00:57:27.302 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.298+0000 7f2fda59c700 1 --2- 172.21.15.115:0/2937298011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 0x7f2fdc0cb880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009840 tx=0x7f2fd0009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=71f0c638a509f2f3 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:27.302 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.298+0000 7f2fd9d9b700 1 -- 172.21.15.115:0/2937298011 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0011830 con 0x7f2fdc15dd30 2023-03-25T00:57:27.302 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.298+0000 7f2fd9d9b700 1 -- 172.21.15.115:0/2937298011 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd0011990 con 0x7f2fdc15dd30 2023-03-25T00:57:27.302 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.298+0000 7f2fd9d9b700 1 -- 172.21.15.115:0/2937298011 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd0011d20 con 0x7f2fdc15dd30 2023-03-25T00:57:27.305 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.299+0000 7f2fe11b5700 1 -- 172.21.15.115:0/2937298011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 msgr2=0x7f2fdc0cb880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:27.305 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.299+0000 7f2fe11b5700 1 --2- 172.21.15.115:0/2937298011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 0x7f2fdc0cb880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009840 tx=0x7f2fd0009b50 comp rx=0 tx=0).stop 2023-03-25T00:57:27.305 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.299+0000 7f2fe11b5700 1 -- 172.21.15.115:0/2937298011 shutdown_connections 2023-03-25T00:57:27.305 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.299+0000 7f2fe11b5700 1 --2- 172.21.15.115:0/2937298011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 0x7f2fdc0cb880 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:27.305 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.299+0000 7f2fe11b5700 1 -- 172.21.15.115:0/2937298011 >> 172.21.15.115:0/2937298011 conn(0x7f2fdc0d0bc0 msgr2=0x7f2fdc0b1b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:27.305 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.299+0000 7f2fe11b5700 1 -- 172.21.15.115:0/2937298011 shutdown_connections 2023-03-25T00:57:27.306 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.299+0000 7f2fe11b5700 1 -- 172.21.15.115:0/2937298011 wait complete. 2023-03-25T00:57:27.306 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.300+0000 7f2fe11b5700 1 Processor -- start 2023-03-25T00:57:27.306 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.300+0000 7f2fe11b5700 1 -- start start 2023-03-25T00:57:27.306 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fe11b5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc071010 0x7f2fdc147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:27.306 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fe11b5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fdc15c900 0x7f2fdc0ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:27.306 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fe11b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 0x7f2fdc147260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:27.306 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fe11b5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc0c8270 con 0x7f2fdc071010 2023-03-25T00:57:27.307 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fe11b5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc148200 con 0x7f2fdc15dd30 2023-03-25T00:57:27.307 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fe11b5700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc0c75f0 con 0x7f2fdc15c900 2023-03-25T00:57:27.307 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fdad9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc071010 0x7f2fdc147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:27.307 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fdb59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 0x7f2fdc147260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:27.307 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fdad9d700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc071010 0x7f2fdc147cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60708/0 (socket says 172.21.15.115:60708) 2023-03-25T00:57:27.308 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fdad9d700 1 -- 172.21.15.115:0/1444884359 learned_addr learned my addr 172.21.15.115:0/1444884359 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:27.308 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.301+0000 7f2fda59c700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fdc15c900 0x7f2fdc0ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:27.308 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fdb59e700 1 -- 172.21.15.115:0/1444884359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fdc15c900 msgr2=0x7f2fdc0ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:27.308 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fdb59e700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fdc15c900 0x7f2fdc0ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:27.308 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fdb59e700 1 -- 172.21.15.115:0/1444884359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc071010 msgr2=0x7f2fdc147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:27.308 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fdb59e700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc071010 0x7f2fdc147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:27.308 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fdb59e700 1 -- 172.21.15.115:0/1444884359 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fd00094a0 con 0x7f2fdc15dd30 2023-03-25T00:57:27.309 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fdad9d700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc071010 0x7f2fdc147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:27.309 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fdb59e700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 0x7f2fdc147260 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400ca80 tx=0x7f2fd400cd90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:27.309 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fc37fe700 1 -- 172.21.15.115:0/1444884359 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd400a9e0 con 0x7f2fdc15dd30 2023-03-25T00:57:27.309 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fc37fe700 1 -- 172.21.15.115:0/1444884359 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd400ab40 con 0x7f2fdc15dd30 2023-03-25T00:57:27.309 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fc37fe700 1 -- 172.21.15.115:0/1444884359 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd4012400 con 0x7f2fdc15dd30 2023-03-25T00:57:27.309 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc15ccd0 con 0x7f2fdc15dd30 2023-03-25T00:57:27.309 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc0b3eb0 con 0x7f2fdc15dd30 2023-03-25T00:57:27.309 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fc37fe700 1 -- 172.21.15.115:0/1444884359 <== mon.1 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2fd4012560 con 0x7f2fdc15dd30 2023-03-25T00:57:27.310 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.302+0000 7f2fc37fe700 1 -- 172.21.15.115:0/1444884359 <== mon.1 v2:172.21.15.121:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f2fd401cb60 con 0x7f2fdc15dd30 2023-03-25T00:57:27.310 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.303+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc0cbdc0 con 0x7f2fdc15dd30 2023-03-25T00:57:27.310 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.305+0000 7f2fc37fe700 1 -- 172.21.15.115:0/1444884359 <== mon.1 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f2fd4021060 con 0x7f2fdc15dd30 2023-03-25T00:57:27.377 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.375+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fdc074580 con 0x7f2fdc15dd30 2023-03-25T00:57:27.379 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.376+0000 7f2fc37fe700 1 -- 172.21.15.115:0/1444884359 <== mon.1 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f2fd4017070 con 0x7f2fdc15dd30 2023-03-25T00:57:27.379 INFO:teuthology.misc.health.smithi115.stdout: 2023-03-25T00:57:27.379 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":10,"fsid":"9c1ea6b6-7c28-4560-af27-3b583d106388","created":"2023-03-25T00:57:23.615145+0000","modified":"2023-03-25T00:57:26.100484+0000","last_up_change":"0.000000","last_in_change":"2023-03-25T00:57:26.100484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6e324a3-c8ad-459c-bb1e-256d32a317db","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"009b7c98-0c48-49f2-98a1-e8d7afe9af58","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b4056d77-ca36-481f-9779-017fb707bda5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"2c09166f-791d-42cf-9ef4-4746592f74e2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2caea232-d588-4aac-bde0-39dae45860ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"cefc141f-68fb-4be4-957b-85ce0049ada5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"325c5ead-d0cd-45e2-b16c-d967f661d738","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"274ccf94-d180-4a8c-a0c3-f620f738252c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-25T00:57:27.380 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 msgr2=0x7f2fdc147260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:27.380 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 0x7f2fdc147260 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f2fd400ca80 tx=0x7f2fd400cd90 comp rx=0 tx=0).stop 2023-03-25T00:57:27.380 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 shutdown_connections 2023-03-25T00:57:27.380 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2fdc15dd30 0x7f2fdc147260 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:27.381 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f2fdc071010 0x7f2fdc147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:27.381 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 --2- 172.21.15.115:0/1444884359 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f2fdc15c900 0x7f2fdc0ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:27.381 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 >> 172.21.15.115:0/1444884359 conn(0x7f2fdc0d0bc0 msgr2=0x7f2fdc0b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:27.381 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 shutdown_connections 2023-03-25T00:57:27.381 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:27.377+0000 7f2fe11b5700 1 -- 172.21.15.115:0/1444884359 wait complete. 2023-03-25T00:57:27.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-25T00:57:27.478 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:27.476+0000 7f9bea30b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-25T00:57:27.481 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:27.479+0000 7fa7398e6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-25T00:57:27.695 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:27.694+0000 7f9bea30b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-25T00:57:27.697 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:27.696+0000 7fa7398e6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-25T00:57:27.849 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:27.847+0000 7f9bea30b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-25T00:57:27.851 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:27.849+0000 7fa7398e6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-25T00:57:27.963 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:27.961+0000 7f9bea30b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-25T00:57:27.966 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:27.964+0000 7fa7398e6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-25T00:57:28.110 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:28.108+0000 7f9bea30b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-25T00:57:28.112 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:28.109+0000 7fa7398e6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-25T00:57:28.380 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:28.379+0000 7f9bea30b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-25T00:57:28.382 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:28.380+0000 7fa7398e6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-25T00:57:28.565 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:28.563+0000 7f9bea30b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-25T00:57:28.566 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:28.564+0000 7fa7398e6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-25T00:57:29.147 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:29.145+0000 7fa7398e6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-25T00:57:29.147 INFO:tasks.ceph.mgr.x.smithi121.stderr:2023-03-25T00:57:29.146+0000 7f9bea30b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-25T00:57:30.014 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-03-25T00:57:30.012+0000 7fdcd06e5640 -1 osd.2 0 log_to_monitors true 2023-03-25T00:57:30.170 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:30.168+0000 7fa6f1b8b700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-03-25T00:57:30.172 INFO:tasks.ceph.mgr.y.smithi115.stderr:2023-03-25T00:57:30.171+0000 7fa6f1b8b700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-03-25T00:57:30.256 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-03-25T00:57:30.255+0000 7ff4c11c4640 -1 osd.6 0 log_to_monitors true 2023-03-25T00:57:30.267 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-03-25T00:57:30.265+0000 7f11fc7d3640 -1 osd.4 0 log_to_monitors true 2023-03-25T00:57:30.273 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-03-25T00:57:30.272+0000 7fdebf5cf640 -1 osd.5 0 log_to_monitors true 2023-03-25T00:57:30.278 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-03-25T00:57:30.277+0000 7f832ae48640 -1 osd.3 0 log_to_monitors true 2023-03-25T00:57:30.288 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-03-25T00:57:30.286+0000 7fce256d6640 -1 osd.0 0 log_to_monitors true 2023-03-25T00:57:30.299 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-03-25T00:57:30.298+0000 7f71331cd640 -1 osd.7 0 log_to_monitors true 2023-03-25T00:57:30.301 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-03-25T00:57:30.300+0000 7f7f65345640 -1 osd.1 0 log_to_monitors true 2023-03-25T00:57:33.269 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-03-25T00:57:33.266+0000 7fce19a8e700 -1 osd.0 0 waiting for initial osdmap 2023-03-25T00:57:33.285 INFO:tasks.ceph.osd.0.smithi115.stderr:2023-03-25T00:57:33.283+0000 7fce11668700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-25T00:57:33.337 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-03-25T00:57:33.334+0000 7fdeb3987700 -1 osd.5 0 waiting for initial osdmap 2023-03-25T00:57:33.357 INFO:tasks.ceph.osd.5.smithi121.stderr:2023-03-25T00:57:33.356+0000 7fdeab561700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-25T00:57:33.378 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-03-25T00:57:33.376+0000 7f831f200700 -1 osd.3 0 waiting for initial osdmap 2023-03-25T00:57:33.383 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-03-25T00:57:33.382+0000 7ff4b557c700 -1 osd.6 0 waiting for initial osdmap 2023-03-25T00:57:33.389 INFO:tasks.ceph.osd.3.smithi115.stderr:2023-03-25T00:57:33.387+0000 7f8316dda700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-25T00:57:33.393 INFO:tasks.ceph.osd.6.smithi121.stderr:2023-03-25T00:57:33.392+0000 7ff4ad156700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-25T00:57:33.400 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-03-25T00:57:33.399+0000 7f7127585700 -1 osd.7 0 waiting for initial osdmap 2023-03-25T00:57:33.401 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-03-25T00:57:33.400+0000 7f11f0b8b700 -1 osd.4 0 waiting for initial osdmap 2023-03-25T00:57:33.411 INFO:tasks.ceph.osd.4.smithi121.stderr:2023-03-25T00:57:33.410+0000 7f11e8765700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-25T00:57:33.412 INFO:tasks.ceph.osd.7.smithi121.stderr:2023-03-25T00:57:33.410+0000 7f711f15f700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-25T00:57:33.430 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-03-25T00:57:33.428+0000 7fdcc4a9d700 -1 osd.2 0 waiting for initial osdmap 2023-03-25T00:57:33.442 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-03-25T00:57:33.440+0000 7f7f596fd700 -1 osd.1 0 waiting for initial osdmap 2023-03-25T00:57:33.444 INFO:tasks.ceph.osd.2.smithi115.stderr:2023-03-25T00:57:33.443+0000 7fdcbc677700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-25T00:57:33.452 INFO:tasks.ceph.osd.1.smithi115.stderr:2023-03-25T00:57:33.451+0000 7f7f512d7700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-25T00:57:34.226 DEBUG:teuthology.orchestra.run.smithi115:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-25T00:57:34.351 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.349+0000 7ff8b2034700 1 Processor -- start 2023-03-25T00:57:34.351 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.349+0000 7ff8b2034700 1 -- start start 2023-03-25T00:57:34.353 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8b2034700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff8ac0cbdb0 0x7ff8ac15de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.353 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8b2034700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cf080 0x7ff8ac159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.353 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8b2034700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8ac15d990 con 0x7ff8ac15a400 2023-03-25T00:57:34.353 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8b2034700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8ac15aea0 con 0x7ff8ac0cbdb0 2023-03-25T00:57:34.353 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8b2034700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac15a8e0 con 0x7ff8ac0cf080 2023-03-25T00:57:34.353 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8ab7fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff8ac0cbdb0 0x7ff8ac15de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.354 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8aaffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cf080 0x7ff8ac159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.354 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8ab7fe700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7ff8ac0cbdb0 0x7ff8ac15de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.115:45220/0 (socket says 172.21.15.115:45220) 2023-03-25T00:57:34.354 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8aaffd700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cf080 0x7ff8ac159ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.115:45028/0 (socket says 172.21.15.115:45028) 2023-03-25T00:57:34.354 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8ab7fe700 1 -- 172.21.15.115:0/2508477707 learned_addr learned my addr 172.21.15.115:0/2508477707 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:34.354 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.350+0000 7ff8abfff700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7ff8ac15a400 0x7ff8ac071050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.115:48830/0 (socket says 172.21.15.115:48830) 2023-03-25T00:57:34.354 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8ab7fe700 1 -- 172.21.15.115:0/2508477707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cf080 msgr2=0x7ff8ac159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:34.354 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8ab7fe700 1 --2- 172.21.15.115:0/2508477707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cf080 0x7ff8ac159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.355 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8ab7fe700 1 -- 172.21.15.115:0/2508477707 >> v1:172.21.15.121:6789/0 conn(0x7ff8ac15a400 legacy=0x7ff8ac071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.355 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8ab7fe700 1 -- 172.21.15.115:0/2508477707 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8ac15abc0 con 0x7ff8ac0cbdb0 2023-03-25T00:57:34.355 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8aaffd700 1 --2- 172.21.15.115:0/2508477707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cf080 0x7ff8ac159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:34.355 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8ab7fe700 1 --2- 172.21.15.115:0/2508477707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac0cbdb0 0x7ff8ac15de20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00bb90 tx=0x7ff89c00bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25bd3dd03f55f822 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:34.355 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8aa7fc700 1 -- 172.21.15.115:0/2508477707 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c00dd30 con 0x7ff8ac0cbdb0 2023-03-25T00:57:34.355 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8aa7fc700 1 -- 172.21.15.115:0/2508477707 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89c00de90 con 0x7ff8ac0cbdb0 2023-03-25T00:57:34.355 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8aa7fc700 1 -- 172.21.15.115:0/2508477707 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89c010690 con 0x7ff8ac0cbdb0 2023-03-25T00:57:34.357 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8b2034700 1 -- 172.21.15.115:0/2508477707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac0cbdb0 msgr2=0x7ff8ac15de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.358 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8b2034700 1 --2- 172.21.15.115:0/2508477707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac0cbdb0 0x7ff8ac15de20 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00bb90 tx=0x7ff89c00bea0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.358 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8b2034700 1 -- 172.21.15.115:0/2508477707 shutdown_connections 2023-03-25T00:57:34.358 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8b2034700 1 --2- 172.21.15.115:0/2508477707 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac0cbdb0 0x7ff8ac15de20 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.358 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8b2034700 1 --2- 172.21.15.115:0/2508477707 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cf080 0x7ff8ac159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.358 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8b2034700 1 -- 172.21.15.115:0/2508477707 >> 172.21.15.115:0/2508477707 conn(0x7ff8ac0cfef0 msgr2=0x7ff8ac0d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:34.358 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8b2034700 1 -- 172.21.15.115:0/2508477707 shutdown_connections 2023-03-25T00:57:34.358 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.351+0000 7ff8b2034700 1 -- 172.21.15.115:0/2508477707 wait complete. 2023-03-25T00:57:34.359 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.353+0000 7ff8b2034700 1 Processor -- start 2023-03-25T00:57:34.359 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.353+0000 7ff8b2034700 1 -- start start 2023-03-25T00:57:34.359 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.353+0000 7ff8b2034700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cbdb0 0x7ff8ac0c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.359 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.353+0000 7ff8b2034700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8ac0cf080 0x7ff8ac0af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.359 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8b2034700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac15a400 0x7ff8ac0ca900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.359 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8b2034700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0b1350 con 0x7ff8ac0cf080 2023-03-25T00:57:34.359 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8b2034700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0b2d20 con 0x7ff8ac15a400 2023-03-25T00:57:34.359 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8b2034700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8ac0b01f0 con 0x7ff8ac0cbdb0 2023-03-25T00:57:34.360 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8ac0cf080 0x7ff8ac0af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.360 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8ac0cf080 0x7ff8ac0af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60930/0 (socket says 172.21.15.115:60930) 2023-03-25T00:57:34.360 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 -- 172.21.15.115:0/1559376969 learned_addr learned my addr 172.21.15.115:0/1559376969 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:34.360 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8abfff700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac15a400 0x7ff8ac0ca900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.360 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8ab7fe700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cbdb0 0x7ff8ac0c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.360 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 -- 172.21.15.115:0/1559376969 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cbdb0 msgr2=0x7ff8ac0c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.360 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cbdb0 0x7ff8ac0c2e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.361 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 -- 172.21.15.115:0/1559376969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac15a400 msgr2=0x7ff8ac0ca900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.361 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac15a400 0x7ff8ac0ca900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.361 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 -- 172.21.15.115:0/1559376969 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00094a0 con 0x7ff8ac0cf080 2023-03-25T00:57:34.361 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8aaffd700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8ac0cf080 0x7ff8ac0af750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000ebe0 tx=0x7ff8a0009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:34.361 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff893fff700 1 -- 172.21.15.115:0/1559376969 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a000c890 con 0x7ff8ac0cf080 2023-03-25T00:57:34.361 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff893fff700 1 -- 172.21.15.115:0/1559376969 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a000c9f0 con 0x7ff8ac0cf080 2023-03-25T00:57:34.361 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89c00b7f0 con 0x7ff8ac0cf080 2023-03-25T00:57:34.361 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff893fff700 1 -- 172.21.15.115:0/1559376969 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8a000cda0 con 0x7ff8ac0cf080 2023-03-25T00:57:34.362 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.354+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8ac0cc180 con 0x7ff8ac0cf080 2023-03-25T00:57:34.362 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.355+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8ac15d620 con 0x7ff8ac0cf080 2023-03-25T00:57:34.362 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.356+0000 7ff893fff700 1 -- 172.21.15.115:0/1559376969 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7ff8a0013470 con 0x7ff8ac0cf080 2023-03-25T00:57:34.362 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.356+0000 7ff893fff700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7ff898069d50 0x7ff89806c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.363 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.356+0000 7ff893fff700 1 -- 172.21.15.115:0/1559376969 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7ff8a0087780 con 0x7ff8ac0cf080 2023-03-25T00:57:34.363 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.356+0000 7ff8ab7fe700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7ff898069d50 0x7ff89806c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.363 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.356+0000 7ff8ab7fe700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7ff898069d50 0x7ff89806c200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00bee0 tx=0x7ff89c013000 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:34.363 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.361+0000 7ff893fff700 1 -- 172.21.15.115:0/1559376969 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7ff8a0053660 con 0x7ff8ac0cf080 2023-03-25T00:57:34.557 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.555+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8ac15e360 con 0x7ff8ac0cf080 2023-03-25T00:57:34.558 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.556+0000 7ff893fff700 1 -- 172.21.15.115:0/1559376969 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+10488 (secure 0 0 0) 0x7ff8a0056c80 con 0x7ff8ac0cf080 2023-03-25T00:57:34.558 INFO:teuthology.misc.health.smithi115.stdout: 2023-03-25T00:57:34.558 INFO:teuthology.misc.health.smithi115.stdout:{"epoch":13,"fsid":"9c1ea6b6-7c28-4560-af27-3b583d106388","created":"2023-03-25T00:57:23.615145+0000","modified":"2023-03-25T00:57:34.273020+0000","last_up_change":"2023-03-25T00:57:34.273020+0000","last_in_change":"2023-03-25T00:57:26.100484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b6e324a3-c8ad-459c-bb1e-256d32a317db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":97654},{"type":"v1","addr":"172.21.15.115:6817","nonce":97654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":97654},{"type":"v1","addr":"172.21.15.115:6819","nonce":97654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":97654},{"type":"v1","addr":"172.21.15.115:6823","nonce":97654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":97654},{"type":"v1","addr":"172.21.15.115:6821","nonce":97654}]},"public_addr":"172.21.15.115:6817/97654","cluster_addr":"172.21.15.115:6819/97654","heartbeat_back_addr":"172.21.15.115:6823/97654","heartbeat_front_addr":"172.21.15.115:6821/97654","state":["exists","up"]},{"osd":1,"uuid":"009b7c98-0c48-49f2-98a1-e8d7afe9af58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":97652},{"type":"v1","addr":"172.21.15.115:6809","nonce":97652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":97652},{"type":"v1","addr":"172.21.15.115:6811","nonce":97652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":97652},{"type":"v1","addr":"172.21.15.115:6815","nonce":97652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":97652},{"type":"v1","addr":"172.21.15.115:6813","nonce":97652}]},"public_addr":"172.21.15.115:6809/97652","cluster_addr":"172.21.15.115:6811/97652","heartbeat_back_addr":"172.21.15.115:6815/97652","heartbeat_front_addr":"172.21.15.115:6813/97652","state":["exists","up"]},{"osd":2,"uuid":"b4056d77-ca36-481f-9779-017fb707bda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":97653},{"type":"v1","addr":"172.21.15.115:6801","nonce":97653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":97653},{"type":"v1","addr":"172.21.15.115:6803","nonce":97653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":97653},{"type":"v1","addr":"172.21.15.115:6807","nonce":97653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":97653},{"type":"v1","addr":"172.21.15.115:6805","nonce":97653}]},"public_addr":"172.21.15.115:6801/97653","cluster_addr":"172.21.15.115:6803/97653","heartbeat_back_addr":"172.21.15.115:6807/97653","heartbeat_front_addr":"172.21.15.115:6805/97653","state":["exists","up"]},{"osd":3,"uuid":"2c09166f-791d-42cf-9ef4-4746592f74e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":97666},{"type":"v1","addr":"172.21.15.115:6825","nonce":97666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":97666},{"type":"v1","addr":"172.21.15.115:6827","nonce":97666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":97666},{"type":"v1","addr":"172.21.15.115:6831","nonce":97666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":97666},{"type":"v1","addr":"172.21.15.115:6829","nonce":97666}]},"public_addr":"172.21.15.115:6825/97666","cluster_addr":"172.21.15.115:6827/97666","heartbeat_back_addr":"172.21.15.115:6831/97666","heartbeat_front_addr":"172.21.15.115:6829/97666","state":["exists","up"]},{"osd":4,"uuid":"2caea232-d588-4aac-bde0-39dae45860ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":97848},{"type":"v1","addr":"172.21.15.121:6817","nonce":97848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":97848},{"type":"v1","addr":"172.21.15.121:6819","nonce":97848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":97848},{"type":"v1","addr":"172.21.15.121:6823","nonce":97848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":97848},{"type":"v1","addr":"172.21.15.121:6821","nonce":97848}]},"public_addr":"172.21.15.121:6817/97848","cluster_addr":"172.21.15.121:6819/97848","heartbeat_back_addr":"172.21.15.121:6823/97848","heartbeat_front_addr":"172.21.15.121:6821/97848","state":["exists","up"]},{"osd":5,"uuid":"cefc141f-68fb-4be4-957b-85ce0049ada5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":97844},{"type":"v1","addr":"172.21.15.121:6801","nonce":97844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":97844},{"type":"v1","addr":"172.21.15.121:6803","nonce":97844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":97844},{"type":"v1","addr":"172.21.15.121:6807","nonce":97844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":97844},{"type":"v1","addr":"172.21.15.121:6805","nonce":97844}]},"public_addr":"172.21.15.121:6801/97844","cluster_addr":"172.21.15.121:6803/97844","heartbeat_back_addr":"172.21.15.121:6807/97844","heartbeat_front_addr":"172.21.15.121:6805/97844","state":["exists","up"]},{"osd":6,"uuid":"325c5ead-d0cd-45e2-b16c-d967f661d738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":97845},{"type":"v1","addr":"172.21.15.121:6809","nonce":97845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":97845},{"type":"v1","addr":"172.21.15.121:6811","nonce":97845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":97845},{"type":"v1","addr":"172.21.15.121:6815","nonce":97845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":97845},{"type":"v1","addr":"172.21.15.121:6813","nonce":97845}]},"public_addr":"172.21.15.121:6809/97845","cluster_addr":"172.21.15.121:6811/97845","heartbeat_back_addr":"172.21.15.121:6815/97845","heartbeat_front_addr":"172.21.15.121:6813/97845","state":["exists","up"]},{"osd":7,"uuid":"274ccf94-d180-4a8c-a0c3-f620f738252c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":97871},{"type":"v1","addr":"172.21.15.121:6825","nonce":97871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":97871},{"type":"v1","addr":"172.21.15.121:6827","nonce":97871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":97871},{"type":"v1","addr":"172.21.15.121:6831","nonce":97871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":97871},{"type":"v1","addr":"172.21.15.121:6829","nonce":97871}]},"public_addr":"172.21.15.121:6825/97871","cluster_addr":"172.21.15.121:6827/97871","heartbeat_back_addr":"172.21.15.121:6831/97871","heartbeat_front_addr":"172.21.15.121:6829/97871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-25T00:57:34.560 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7ff898069d50 msgr2=0x7ff89806c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.560 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7ff898069d50 0x7ff89806c200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff89c00bee0 tx=0x7ff89c013000 comp rx=0 tx=0).stop 2023-03-25T00:57:34.560 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8ac0cf080 msgr2=0x7ff8ac0af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.561 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8ac0cf080 0x7ff8ac0af750 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff8a000ebe0 tx=0x7ff8a0009e60 comp rx=0 tx=0).stop 2023-03-25T00:57:34.561 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 shutdown_connections 2023-03-25T00:57:34.561 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7ff898069d50 0x7ff89806c200 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.561 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8ac15a400 0x7ff8ac0ca900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.561 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7ff8ac0cbdb0 0x7ff8ac0c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.561 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 --2- 172.21.15.115:0/1559376969 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7ff8ac0cf080 0x7ff8ac0af750 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.561 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 >> 172.21.15.115:0/1559376969 conn(0x7ff8ac0cfef0 msgr2=0x7ff8ac159490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:34.562 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 shutdown_connections 2023-03-25T00:57:34.562 INFO:teuthology.misc.health.smithi115.stderr:2023-03-25T00:57:34.558+0000 7ff8b2034700 1 -- 172.21.15.115:0/1559376969 wait complete. 2023-03-25T00:57:34.571 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-03-25T00:57:34.571 INFO:tasks.ceph:Creating RBD pool 2023-03-25T00:57:34.571 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd pool create rbd 8 2023-03-25T00:57:34.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.689+0000 7f6878463700 1 Processor -- start 2023-03-25T00:57:34.691 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.689+0000 7f6878463700 1 -- start start 2023-03-25T00:57:34.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f6878463700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f687015a620 0x7f68700c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f6878463700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68700b4050 0x7f68700b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.693 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f6878463700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68700c7440 con 0x7f687015a9f0 2023-03-25T00:57:34.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f6878463700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68700b4a30 con 0x7f687015a620 2023-03-25T00:57:34.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f6878463700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68700ca250 con 0x7f68700b4050 2023-03-25T00:57:34.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f68761ff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f687015a620 0x7f68700c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f6876a00700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68700b4050 0x7f68700b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f68761ff700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f687015a620 0x7f68700c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.115:45262/0 (socket says 172.21.15.115:45262) 2023-03-25T00:57:34.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.690+0000 7f68761ff700 1 -- 172.21.15.115:0/2012030540 learned_addr learned my addr 172.21.15.115:0/2012030540 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:34.694 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68751fd700 1 -- 172.21.15.115:0/2012030540 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264318739 0 0) 0x7f68700c7440 con 0x7f687015a9f0 2023-03-25T00:57:34.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68751fd700 1 -- 172.21.15.115:0/2012030540 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f685c0035b0 con 0x7f687015a9f0 2023-03-25T00:57:34.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68761ff700 1 -- 172.21.15.115:0/2012030540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68700b4050 msgr2=0x7f68700b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:34.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68761ff700 1 --2- 172.21.15.115:0/2012030540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68700b4050 0x7f68700b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68761ff700 1 -- 172.21.15.115:0/2012030540 >> v1:172.21.15.121:6789/0 conn(0x7f687015a9f0 legacy=0x7f687015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.695 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68761ff700 1 -- 172.21.15.115:0/2012030540 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687015ef10 con 0x7f687015a620 2023-03-25T00:57:34.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f6876a00700 1 --2- 172.21.15.115:0/2012030540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68700b4050 0x7f68700b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:34.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68761ff700 1 --2- 172.21.15.115:0/2012030540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a620 0x7f68700c6f00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68600098f0 tx=0x7f6860009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9411fc188135cc02 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:34.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68751fd700 1 -- 172.21.15.115:0/2012030540 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860011830 con 0x7f687015a620 2023-03-25T00:57:34.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68751fd700 1 -- 172.21.15.115:0/2012030540 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6860011990 con 0x7f687015a620 2023-03-25T00:57:34.696 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.691+0000 7f68751fd700 1 -- 172.21.15.115:0/2012030540 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860011d20 con 0x7f687015a620 2023-03-25T00:57:34.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.692+0000 7f6878463700 1 -- 172.21.15.115:0/2012030540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a620 msgr2=0x7f68700c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.692+0000 7f6878463700 1 --2- 172.21.15.115:0/2012030540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a620 0x7f68700c6f00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f68600098f0 tx=0x7f6860009c00 comp rx=0 tx=0).stop 2023-03-25T00:57:34.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.692+0000 7f6878463700 1 -- 172.21.15.115:0/2012030540 shutdown_connections 2023-03-25T00:57:34.699 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.692+0000 7f6878463700 1 --2- 172.21.15.115:0/2012030540 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f68700b4050 0x7f68700b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.692+0000 7f6878463700 1 --2- 172.21.15.115:0/2012030540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a620 0x7f68700c6f00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.692+0000 7f6878463700 1 -- 172.21.15.115:0/2012030540 >> 172.21.15.115:0/2012030540 conn(0x7f68700d0b50 msgr2=0x7f68700cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:34.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.692+0000 7f6878463700 1 -- 172.21.15.115:0/2012030540 shutdown_connections 2023-03-25T00:57:34.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.692+0000 7f6878463700 1 -- 172.21.15.115:0/2012030540 wait complete. 2023-03-25T00:57:34.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.693+0000 7f6878463700 1 Processor -- start 2023-03-25T00:57:34.700 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.693+0000 7f6878463700 1 -- start start 2023-03-25T00:57:34.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6878463700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68700b4050 0x7f68700c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6878463700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f687015a620 0x7f68700c7820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6878463700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a9f0 0x7f68701f3d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6878463700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68701f12f0 con 0x7f68700b4050 2023-03-25T00:57:34.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6878463700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68701f06e0 con 0x7f687015a9f0 2023-03-25T00:57:34.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6878463700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68701f09c0 con 0x7f687015a620 2023-03-25T00:57:34.701 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68700b4050 0x7f68700c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68700b4050 0x7f68700c72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60970/0 (socket says 172.21.15.115:60970) 2023-03-25T00:57:34.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 -- 172.21.15.115:0/1688809770 learned_addr learned my addr 172.21.15.115:0/1688809770 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:34.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6876a00700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a9f0 0x7f68701f3d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68759fe700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f687015a620 0x7f68700c7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 -- 172.21.15.115:0/1688809770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f687015a620 msgr2=0x7f68700c7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f687015a620 0x7f68700c7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.702 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 -- 172.21.15.115:0/1688809770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a9f0 msgr2=0x7f68701f3d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:34.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a9f0 0x7f68701f3d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:34.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 -- 172.21.15.115:0/1688809770 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6868009a70 con 0x7f68700b4050 2023-03-25T00:57:34.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f68761ff700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68700b4050 0x7f68700c72e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6860000c00 tx=0x7f6860004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:34.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6866ffd700 1 -- 172.21.15.115:0/1688809770 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6860004480 con 0x7f68700b4050 2023-03-25T00:57:34.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6866ffd700 1 -- 172.21.15.115:0/1688809770 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f686001f070 con 0x7f68700b4050 2023-03-25T00:57:34.703 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6860009550 con 0x7f68700b4050 2023-03-25T00:57:34.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6866ffd700 1 -- 172.21.15.115:0/1688809770 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68600038c0 con 0x7f68700b4050 2023-03-25T00:57:34.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.694+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68700ca480 con 0x7f68700b4050 2023-03-25T00:57:34.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.695+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f687015f1f0 con 0x7f68700b4050 2023-03-25T00:57:34.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.696+0000 7f6866ffd700 1 -- 172.21.15.115:0/1688809770 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7f6860021030 con 0x7f68700b4050 2023-03-25T00:57:34.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.696+0000 7f6866ffd700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f6854069d50 0x7f685406c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:34.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.696+0000 7f6866ffd700 1 -- 172.21.15.115:0/1688809770 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 4471+0+0 (secure 0 0 0) 0x7f6860088ca0 con 0x7f68700b4050 2023-03-25T00:57:34.704 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.696+0000 7f68759fe700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f6854069d50 0x7f685406c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:34.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.696+0000 7f68759fe700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f6854069d50 0x7f685406c200 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f686c007fd0 tx=0x7f686c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:34.705 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.700+0000 7f6866ffd700 1 -- 172.21.15.115:0/1688809770 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f6860054c60 con 0x7f68700b4050 2023-03-25T00:57:34.892 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:34.890+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f68700ca760 con 0x7f68700b4050 2023-03-25T00:57:35.282 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.280+0000 7f6866ffd700 1 -- 172.21.15.115:0/1688809770 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v14) v1 ==== 109+0+0 (secure 0 0 0) 0x7f68600150a0 con 0x7f68700b4050 2023-03-25T00:57:35.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f6854069d50 msgr2=0x7f685406c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:35.285 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f6854069d50 0x7f685406c200 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f686c007fd0 tx=0x7f686c009040 comp rx=0 tx=0).stop 2023-03-25T00:57:35.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68700b4050 msgr2=0x7f68700c72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:35.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68700b4050 0x7f68700c72e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6860000c00 tx=0x7f6860004750 comp rx=0 tx=0).stop 2023-03-25T00:57:35.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 shutdown_connections 2023-03-25T00:57:35.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f6854069d50 0x7f685406c200 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f687015a9f0 0x7f68701f3d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f68700b4050 0x7f68700c72e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 --2- 172.21.15.115:0/1688809770 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f687015a620 0x7f68700c7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 >> 172.21.15.115:0/1688809770 conn(0x7f68700d0b50 msgr2=0x7f68700c9940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:35.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 shutdown_connections 2023-03-25T00:57:35.287 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.283+0000 7f6878463700 1 -- 172.21.15.115:0/1688809770 wait complete. 2023-03-25T00:57:35.287 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'rbd' created 2023-03-25T00:57:35.298 DEBUG:teuthology.orchestra.run.smithi115:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-03-25T00:57:35.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.418+0000 7fa5c5d85700 1 Processor -- start 2023-03-25T00:57:35.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.418+0000 7fa5c5d85700 1 -- start start 2023-03-25T00:57:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5c5d85700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa5c00d2210 0x7fa5c015d990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5c5d85700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d25e0 0x7fa5c006c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5c5d85700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5c00ce6d0 con 0x7fa5c00ceb60 2023-03-25T00:57:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5c5d85700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5c00cfb80 con 0x7fa5c00d2210 2023-03-25T00:57:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5c5d85700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00cdda0 con 0x7fa5c00d25e0 2023-03-25T00:57:35.423 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5bf7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa5c00d2210 0x7fa5c015d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5bf7fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fa5c00d2210 0x7fa5c015d990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60980/0 (socket says 172.21.15.115:60980) 2023-03-25T00:57:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5bf7fe700 1 -- 172.21.15.115:0/309598118 learned_addr learned my addr 172.21.15.115:0/309598118 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5beffd700 1 --2- 172.21.15.115:0/309598118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d25e0 0x7fa5c006c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5bf7fe700 1 -- 172.21.15.115:0/309598118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d25e0 msgr2=0x7fa5c006c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5bf7fe700 1 --2- 172.21.15.115:0/309598118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d25e0 0x7fa5c006c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5bf7fe700 1 -- 172.21.15.115:0/309598118 >> v1:172.21.15.121:6789/0 conn(0x7fa5c00ceb60 legacy=0x7fa5c006a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5bf7fe700 1 -- 172.21.15.115:0/309598118 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c00ce080 con 0x7fa5c00d2210 2023-03-25T00:57:35.424 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.419+0000 7fa5beffd700 1 --2- 172.21.15.115:0/309598118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d25e0 0x7fa5c006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5bf7fe700 1 --2- 172.21.15.115:0/309598118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d2210 0x7fa5c015d990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40098f0 tx=0x7fa5b4009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d92d4e34f62a64f server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5be7fc700 1 -- 172.21.15.115:0/309598118 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b4004230 con 0x7fa5c00d2210 2023-03-25T00:57:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5be7fc700 1 -- 172.21.15.115:0/309598118 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b4004390 con 0x7fa5c00d2210 2023-03-25T00:57:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5be7fc700 1 -- 172.21.15.115:0/309598118 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b40036a0 con 0x7fa5c00d2210 2023-03-25T00:57:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5c5d85700 1 -- 172.21.15.115:0/309598118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d2210 msgr2=0x7fa5c015d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/309598118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d2210 0x7fa5c015d990 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa5b40098f0 tx=0x7fa5b4009c00 comp rx=0 tx=0).stop 2023-03-25T00:57:35.425 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5c5d85700 1 -- 172.21.15.115:0/309598118 shutdown_connections 2023-03-25T00:57:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/309598118 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d2210 0x7fa5c015d990 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/309598118 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d25e0 0x7fa5c006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5c5d85700 1 -- 172.21.15.115:0/309598118 >> 172.21.15.115:0/309598118 conn(0x7fa5c00d09e0 msgr2=0x7fa5c00d32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5c5d85700 1 -- 172.21.15.115:0/309598118 shutdown_connections 2023-03-25T00:57:35.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.420+0000 7fa5c5d85700 1 -- 172.21.15.115:0/309598118 wait complete. 2023-03-25T00:57:35.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.422+0000 7fa5c5d85700 1 Processor -- start 2023-03-25T00:57:35.428 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.422+0000 7fa5c5d85700 1 -- start start 2023-03-25T00:57:35.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.422+0000 7fa5c5d85700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5c00ceb60 0x7fa5c00c5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:35.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.422+0000 7fa5c5d85700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d2210 0x7fa5c01e2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:35.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.422+0000 7fa5c5d85700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d25e0 0x7fa5c015b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:35.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.422+0000 7fa5c5d85700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c015a1e0 con 0x7fa5c00d25e0 2023-03-25T00:57:35.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.422+0000 7fa5c5d85700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c015b040 con 0x7fa5c00ceb60 2023-03-25T00:57:35.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.422+0000 7fa5c5d85700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5c00c61b0 con 0x7fa5c00d2210 2023-03-25T00:57:35.429 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d25e0 0x7fa5c015b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:35.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d25e0 0x7fa5c015b4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:60990/0 (socket says 172.21.15.115:60990) 2023-03-25T00:57:35.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 -- 172.21.15.115:0/1888704847 learned_addr learned my addr 172.21.15.115:0/1888704847 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:35.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bf7fe700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5c00ceb60 0x7fa5c00c5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:35.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 -- 172.21.15.115:0/1888704847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d2210 msgr2=0x7fa5c01e2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:35.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d2210 0x7fa5c01e2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 -- 172.21.15.115:0/1888704847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5c00ceb60 msgr2=0x7fa5c00c5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:35.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5c00ceb60 0x7fa5c00c5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:35.430 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 -- 172.21.15.115:0/1888704847 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b4009550 con 0x7fa5c00d25e0 2023-03-25T00:57:35.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bf7fe700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5c00ceb60 0x7fa5c00c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:35.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5bffff700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d25e0 0x7fa5c015b4d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800cbd0 tx=0x7fa5b800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:35.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5a7fff700 1 -- 172.21.15.115:0/1888704847 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800ab70 con 0x7fa5c00d25e0 2023-03-25T00:57:35.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5a7fff700 1 -- 172.21.15.115:0/1888704847 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b800acd0 con 0x7fa5c00d25e0 2023-03-25T00:57:35.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5a7fff700 1 -- 172.21.15.115:0/1888704847 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8012660 con 0x7fa5c00d25e0 2023-03-25T00:57:35.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5c00cdab0 con 0x7fa5c00d25e0 2023-03-25T00:57:35.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.423+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5c00ce360 con 0x7fa5c00d25e0 2023-03-25T00:57:35.431 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.424+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5c006acb0 con 0x7fa5c00d25e0 2023-03-25T00:57:35.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.425+0000 7fa5a7fff700 1 -- 172.21.15.115:0/1888704847 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7fa5b80127c0 con 0x7fa5c00d25e0 2023-03-25T00:57:35.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.425+0000 7fa5a7fff700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fa5ac069d50 0x7fa5ac06c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:35.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.425+0000 7fa5bf7fe700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fa5ac069d50 0x7fa5ac06c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:35.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.425+0000 7fa5a7fff700 1 -- 172.21.15.115:0/1888704847 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 5188+0+0 (secure 0 0 0) 0x7fa5b80883a0 con 0x7fa5c00d25e0 2023-03-25T00:57:35.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.426+0000 7fa5bf7fe700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fa5ac069d50 0x7fa5ac06c200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4009c40 tx=0x7fa5b4008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:35.432 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.430+0000 7fa5a7fff700 1 -- 172.21.15.115:0/1888704847 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7fa5b8054030 con 0x7fa5c00d25e0 2023-03-25T00:57:35.623 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:35.621+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa5c0074580 con 0x7fa5c00d25e0 2023-03-25T00:57:36.286 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.283+0000 7fa5a7fff700 1 -- 172.21.15.115:0/1888704847 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v15) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa5b8057650 con 0x7fa5c00d25e0 2023-03-25T00:57:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fa5ac069d50 msgr2=0x7fa5ac06c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fa5ac069d50 0x7fa5ac06c200 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa5b4009c40 tx=0x7fa5b4008030 comp rx=0 tx=0).stop 2023-03-25T00:57:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d25e0 msgr2=0x7fa5c015b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d25e0 0x7fa5c015b4d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800cbd0 tx=0x7fa5b800cf90 comp rx=0 tx=0).stop 2023-03-25T00:57:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 shutdown_connections 2023-03-25T00:57:36.289 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fa5ac069d50 0x7fa5ac06c200 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5c00ceb60 0x7fa5c00c5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fa5c00d25e0 0x7fa5c015b4d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 --2- 172.21.15.115:0/1888704847 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fa5c00d2210 0x7fa5c01e2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.286+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 >> 172.21.15.115:0/1888704847 conn(0x7fa5c00d09e0 msgr2=0x7fa5c00c6e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.287+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 shutdown_connections 2023-03-25T00:57:36.290 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.287+0000 7fa5c5d85700 1 -- 172.21.15.115:0/1888704847 wait complete. 2023-03-25T00:57:36.291 INFO:teuthology.orchestra.run.smithi115.stderr:enabled application 'rbd' on pool 'rbd' 2023-03-25T00:57:36.306 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-03-25T00:57:36.306 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-25T00:57:36.307 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-25T00:57:36.309 INFO:tasks.ceph.mds.a:Started 2023-03-25T00:57:36.309 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-25T00:57:36.309 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-25T00:57:36.311 INFO:tasks.ceph.mds.c:Started 2023-03-25T00:57:36.311 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-25T00:57:36.312 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-25T00:57:36.314 INFO:tasks.ceph.mds.b:Started 2023-03-25T00:57:36.314 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-25T00:57:36.315 DEBUG:teuthology.orchestra.run.smithi121:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-25T00:57:36.316 INFO:tasks.ceph.mds.d:Started 2023-03-25T00:57:36.317 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-03-25T00:57:36.317 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-25T00:57:36.400 INFO:tasks.ceph.mds.c.smithi115.stdout:starting mds.c at 2023-03-25T00:57:36.400 INFO:tasks.ceph.mds.a.smithi115.stdout:starting mds.a at 2023-03-25T00:57:36.405 INFO:tasks.ceph.mds.d.smithi121.stdout:starting mds.d at 2023-03-25T00:57:36.406 INFO:tasks.ceph.mds.b.smithi121.stdout:starting mds.b at 2023-03-25T00:57:36.406 INFO:tasks.ceph.mds.d.smithi121.stderr:2023-03-25T00:57:36.405+0000 7f76350b4700 -1 PosixStack listen unable to listen on v1:0.0.0.0:6833/0: (98) Address already in use 2023-03-25T00:57:36.466 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.464+0000 7f39dcf7f700 1 Processor -- start 2023-03-25T00:57:36.467 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.464+0000 7f39dcf7f700 1 -- start start 2023-03-25T00:57:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39dcf7f700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f39d80b2b90 0x7f39d80b3010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39dcf7f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80cd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39dcf7f700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39d8151e80 con 0x7f39d806f9e0 2023-03-25T00:57:36.468 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39dcf7f700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39d80d0e90 con 0x7f39d80b2b90 2023-03-25T00:57:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39dcf7f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d80b3550 con 0x7f39d80cd460 2023-03-25T00:57:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39d659c700 1 --1- >> v1:172.21.15.121:6789/0 conn(0x7f39d806f9e0 0x7f39d80cfef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.115:48966/0 (socket says 172.21.15.115:48966) 2023-03-25T00:57:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39d6d9d700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80cd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39d5d9b700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f39d80b2b90 0x7f39d80b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.469 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.465+0000 7f39d659c700 1 -- 172.21.15.115:0/3781219922 learned_addr learned my addr 172.21.15.115:0/3781219922 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d5d9b700 1 -- 172.21.15.115:0/3781219922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 msgr2=0x7f39d80cd890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d5d9b700 1 --2- 172.21.15.115:0/3781219922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80cd890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d5d9b700 1 -- 172.21.15.115:0/3781219922 >> v1:172.21.15.121:6789/0 conn(0x7f39d806f9e0 legacy=0x7f39d80cfef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d5d9b700 1 -- 172.21.15.115:0/3781219922 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d80cc9e0 con 0x7f39d80b2b90 2023-03-25T00:57:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d559a700 1 -- 172.21.15.115:0/3781219922 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255262263 0 0) 0x7f39d8151e80 con 0x7f39d806f9e0 2023-03-25T00:57:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d5d9b700 1 --2- 172.21.15.115:0/3781219922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d80b2b90 0x7f39d80b3010 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009840 tx=0x7f39cc009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2d58030925337dc server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:36.470 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d559a700 1 -- 172.21.15.115:0/3781219922 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc011830 con 0x7f39d80b2b90 2023-03-25T00:57:36.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d559a700 1 -- 172.21.15.115:0/3781219922 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39cc011990 con 0x7f39d80b2b90 2023-03-25T00:57:36.471 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.466+0000 7f39d559a700 1 -- 172.21.15.115:0/3781219922 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc011d20 con 0x7f39d80b2b90 2023-03-25T00:57:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.467+0000 7f39dcf7f700 1 -- 172.21.15.115:0/3781219922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d80b2b90 msgr2=0x7f39d80b3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.467+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/3781219922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d80b2b90 0x7f39d80b3010 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f39cc009840 tx=0x7f39cc009b50 comp rx=0 tx=0).stop 2023-03-25T00:57:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.467+0000 7f39dcf7f700 1 -- 172.21.15.115:0/3781219922 shutdown_connections 2023-03-25T00:57:36.474 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.467+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/3781219922 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80cd890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.467+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/3781219922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d80b2b90 0x7f39d80b3010 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.467+0000 7f39dcf7f700 1 -- 172.21.15.115:0/3781219922 >> 172.21.15.115:0/3781219922 conn(0x7f39d8070270 msgr2=0x7f39d8070640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.467+0000 7f39dcf7f700 1 -- 172.21.15.115:0/3781219922 shutdown_connections 2023-03-25T00:57:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.467+0000 7f39dcf7f700 1 -- 172.21.15.115:0/3781219922 wait complete. 2023-03-25T00:57:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.468+0000 7f39dcf7f700 1 Processor -- start 2023-03-25T00:57:36.475 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.468+0000 7f39dcf7f700 1 -- start start 2023-03-25T00:57:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39dcf7f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d806f9e0 0x7f39d806bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39dcf7f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f39d80b2b90 0x7f39d814b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39dcf7f700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39dcf7f700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d815a400 con 0x7f39d80b2b90 2023-03-25T00:57:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39dcf7f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d80c9730 con 0x7f39d806f9e0 2023-03-25T00:57:36.476 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39dcf7f700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39d814be40 con 0x7f39d80cd460 2023-03-25T00:57:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f39d80b2b90 0x7f39d814b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f39d80b2b90 0x7f39d814b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32838/0 (socket says 172.21.15.115:32838) 2023-03-25T00:57:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 -- 172.21.15.115:0/780402544 learned_addr learned my addr 172.21.15.115:0/780402544 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d6d9d700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.477 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d659c700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d806f9e0 0x7f39d806bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 -- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 msgr2=0x7f39d80c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 -- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d806f9e0 msgr2=0x7f39d806bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d806f9e0 0x7f39d806bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.478 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 -- 172.21.15.115:0/780402544 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39cc0094a0 con 0x7f39d80b2b90 2023-03-25T00:57:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d6d9d700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-25T00:57:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.469+0000 7f39d5d9b700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f39d80b2b90 0x7f39d814b900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0090b0 tx=0x7f39cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.470+0000 7f39c6ffd700 1 -- 172.21.15.115:0/780402544 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc0038a0 con 0x7f39d80b2b90 2023-03-25T00:57:36.479 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.470+0000 7f39c6ffd700 1 -- 172.21.15.115:0/780402544 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39cc01f070 con 0x7f39d80b2b90 2023-03-25T00:57:36.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.470+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d80d08d0 con 0x7f39d80b2b90 2023-03-25T00:57:36.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.470+0000 7f39c6ffd700 1 -- 172.21.15.115:0/780402544 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39cc003c40 con 0x7f39d80b2b90 2023-03-25T00:57:36.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.470+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d80d0600 con 0x7f39d80b2b90 2023-03-25T00:57:36.480 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.471+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d80d0bb0 con 0x7f39d80b2b90 2023-03-25T00:57:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.471+0000 7f39c6ffd700 1 -- 172.21.15.115:0/780402544 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7f39cc026480 con 0x7f39d80b2b90 2023-03-25T00:57:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.471+0000 7f39c6ffd700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f39b8069d50 0x7f39b806c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.472+0000 7f39d659c700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f39b8069d50 0x7f39b806c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.481 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.472+0000 7f39c6ffd700 1 -- 172.21.15.115:0/780402544 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f39cc021070 con 0x7f39d80b2b90 2023-03-25T00:57:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.472+0000 7f39d659c700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f39b8069d50 0x7f39b806c200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f39c8000f60 tx=0x7f39c8007c90 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:36.482 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.476+0000 7f39c6ffd700 1 -- 172.21.15.115:0/780402544 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f39cc0552e0 con 0x7f39d80b2b90 2023-03-25T00:57:36.677 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.675+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39d80735b0 con 0x7f39d80b2b90 2023-03-25T00:57:36.678 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.676+0000 7f39c6ffd700 1 -- 172.21.15.115:0/780402544 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+14185 (secure 0 0 0) 0x7f39cc058900 con 0x7f39d80b2b90 2023-03-25T00:57:36.678 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:57:36.679 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":15,"fsid":"9c1ea6b6-7c28-4560-af27-3b583d106388","created":"2023-03-25T00:57:23.615145+0000","modified":"2023-03-25T00:57:36.280207+0000","last_up_change":"2023-03-25T00:57:34.273020+0000","last_in_change":"2023-03-25T00:57:26.100484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-03-25T00:57:34.908808+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-03-25T00:57:35.207270+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6e324a3-c8ad-459c-bb1e-256d32a317db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":97654},{"type":"v1","addr":"172.21.15.115:6817","nonce":97654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":97654},{"type":"v1","addr":"172.21.15.115:6819","nonce":97654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":97654},{"type":"v1","addr":"172.21.15.115:6823","nonce":97654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":97654},{"type":"v1","addr":"172.21.15.115:6821","nonce":97654}]},"public_addr":"172.21.15.115:6817/97654","cluster_addr":"172.21.15.115:6819/97654","heartbeat_back_addr":"172.21.15.115:6823/97654","heartbeat_front_addr":"172.21.15.115:6821/97654","state":["exists","up"]},{"osd":1,"uuid":"009b7c98-0c48-49f2-98a1-e8d7afe9af58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":97652},{"type":"v1","addr":"172.21.15.115:6809","nonce":97652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":97652},{"type":"v1","addr":"172.21.15.115:6811","nonce":97652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":97652},{"type":"v1","addr":"172.21.15.115:6815","nonce":97652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":97652},{"type":"v1","addr":"172.21.15.115:6813","nonce":97652}]},"public_addr":"172.21.15.115:6809/97652","cluster_addr":"172.21.15.115:6811/97652","heartbeat_back_addr":"172.21.15.115:6815/97652","heartbeat_front_addr":"172.21.15.115:6813/97652","state":["exists","up"]},{"osd":2,"uuid":"b4056d77-ca36-481f-9779-017fb707bda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":97653},{"type":"v1","addr":"172.21.15.115:6801","nonce":97653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":97653},{"type":"v1","addr":"172.21.15.115:6803","nonce":97653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":97653},{"type":"v1","addr":"172.21.15.115:6807","nonce":97653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":97653},{"type":"v1","addr":"172.21.15.115:6805","nonce":97653}]},"public_addr":"172.21.15.115:6801/97653","cluster_addr":"172.21.15.115:6803/97653","heartbeat_back_addr":"172.21.15.115:6807/97653","heartbeat_front_addr":"172.21.15.115:6805/97653","state":["exists","up"]},{"osd":3,"uuid":"2c09166f-791d-42cf-9ef4-4746592f74e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":97666},{"type":"v1","addr":"172.21.15.115:6825","nonce":97666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":97666},{"type":"v1","addr":"172.21.15.115:6827","nonce":97666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":97666},{"type":"v1","addr":"172.21.15.115:6831","nonce":97666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":97666},{"type":"v1","addr":"172.21.15.115:6829","nonce":97666}]},"public_addr":"172.21.15.115:6825/97666","cluster_addr":"172.21.15.115:6827/97666","heartbeat_back_addr":"172.21.15.115:6831/97666","heartbeat_front_addr":"172.21.15.115:6829/97666","state":["exists","up"]},{"osd":4,"uuid":"2caea232-d588-4aac-bde0-39dae45860ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":97848},{"type":"v1","addr":"172.21.15.121:6817","nonce":97848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":97848},{"type":"v1","addr":"172.21.15.121:6819","nonce":97848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":97848},{"type":"v1","addr":"172.21.15.121:6823","nonce":97848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":97848},{"type":"v1","addr":"172.21.15.121:6821","nonce":97848}]},"public_addr":"172.21.15.121:6817/97848","cluster_addr":"172.21.15.121:6819/97848","heartbeat_back_addr":"172.21.15.121:6823/97848","heartbeat_front_addr":"172.21.15.121:6821/97848","state":["exists","up"]},{"osd":5,"uuid":"cefc141f-68fb-4be4-957b-85ce0049ada5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":97844},{"type":"v1","addr":"172.21.15.121:6801","nonce":97844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":97844},{"type":"v1","addr":"172.21.15.121:6803","nonce":97844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":97844},{"type":"v1","addr":"172.21.15.121:6807","nonce":97844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":97844},{"type":"v1","addr":"172.21.15.121:6805","nonce":97844}]},"public_addr":"172.21.15.121:6801/97844","cluster_addr":"172.21.15.121:6803/97844","heartbeat_back_addr":"172.21.15.121:6807/97844","heartbeat_front_addr":"172.21.15.121:6805/97844","state":["exists","up"]},{"osd":6,"uuid":"325c5ead-d0cd-45e2-b16c-d967f661d738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":97845},{"type":"v1","addr":"172.21.15.121:6809","nonce":97845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":97845},{"type":"v1","addr":"172.21.15.121:6811","nonce":97845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":97845},{"type":"v1","addr":"172.21.15.121:6815","nonce":97845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":97845},{"type":"v1","addr":"172.21.15.121:6813","nonce":97845}]},"public_addr":"172.21.15.121:6809/97845","cluster_addr":"172.21.15.121:6811/97845","heartbeat_back_addr":"172.21.15.121:6815/97845","heartbeat_front_addr":"172.21.15.121:6813/97845","state":["exists","up"]},{"osd":7,"uuid":"274ccf94-d180-4a8c-a0c3-f620f738252c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":97871},{"type":"v1","addr":"172.21.15.121:6825","nonce":97871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":97871},{"type":"v1","addr":"172.21.15.121:6827","nonce":97871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":97871},{"type":"v1","addr":"172.21.15.121:6831","nonce":97871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":97871},{"type":"v1","addr":"172.21.15.121:6829","nonce":97871}]},"public_addr":"172.21.15.121:6825/97871","cluster_addr":"172.21.15.121:6827/97871","heartbeat_back_addr":"172.21.15.121:6831/97871","heartbeat_front_addr":"172.21.15.121:6829/97871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.246439+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.328118+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:30.993313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.265418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.261797+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.284125+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.233058+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.330665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-25T00:57:36.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f39b8069d50 msgr2=0x7f39b806c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f39b8069d50 0x7f39b806c200 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f39c8000f60 tx=0x7f39c8007c90 comp rx=0 tx=0).stop 2023-03-25T00:57:36.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f39d80b2b90 msgr2=0x7f39d814b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f39d80b2b90 0x7f39d814b900 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f39cc0090b0 tx=0x7f39cc004750 comp rx=0 tx=0).stop 2023-03-25T00:57:36.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 shutdown_connections 2023-03-25T00:57:36.681 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f39b8069d50 0x7f39b806c200 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d806f9e0 0x7f39d806bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f39d80cd460 0x7f39d80c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 --2- 172.21.15.115:0/780402544 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f39d80b2b90 0x7f39d814b900 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.678+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 >> 172.21.15.115:0/780402544 conn(0x7f39d8070270 msgr2=0x7f39d80b2440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:36.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.679+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 shutdown_connections 2023-03-25T00:57:36.682 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.679+0000 7f39dcf7f700 1 -- 172.21.15.115:0/780402544 wait complete. 2023-03-25T00:57:36.693 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-03-25T00:57:34.908808+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-03-25T00:57:35.207270+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-25T00:57:36.695 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-25T00:57:36.823 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.822+0000 7f98b80ac700 1 Processor -- start 2023-03-25T00:57:36.824 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.822+0000 7f98b80ac700 1 -- start start 2023-03-25T00:57:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.823+0000 7f98b80ac700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f98b015a620 0x7f98b00c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.823+0000 7f98b80ac700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015a9f0 0x7f98b015e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.825 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.823+0000 7f98b80ac700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b00c7440 con 0x7f98b015e8e0 2023-03-25T00:57:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.823+0000 7f98b80ac700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b00ca250 con 0x7f98b015a620 2023-03-25T00:57:36.826 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.823+0000 7f98b80ac700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b015ef10 con 0x7f98b015a9f0 2023-03-25T00:57:36.827 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5e48700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f98b015a620 0x7f98b00c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5647700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015a9f0 0x7f98b015e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5e48700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f98b015a620 0x7f98b00c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.115:45380/0 (socket says 172.21.15.115:45380) 2023-03-25T00:57:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5e48700 1 -- 172.21.15.115:0/2397002403 learned_addr learned my addr 172.21.15.115:0/2397002403 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b6649700 1 --1- 172.21.15.115:0/2397002403 >> v1:172.21.15.121:6789/0 conn(0x7f98b015e8e0 0x7f98b00b4050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.121:6789/0 says I am v1:172.21.15.115:48990/0 (socket says 172.21.15.115:48990) 2023-03-25T00:57:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5e48700 1 -- 172.21.15.115:0/2397002403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015a9f0 msgr2=0x7f98b015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:36.828 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5e48700 1 --2- 172.21.15.115:0/2397002403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015a9f0 0x7f98b015e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5e48700 1 -- 172.21.15.115:0/2397002403 >> v1:172.21.15.121:6789/0 conn(0x7f98b015e8e0 legacy=0x7f98b00b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5e48700 1 -- 172.21.15.115:0/2397002403 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b015f1f0 con 0x7f98b015a620 2023-03-25T00:57:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b4e46700 1 -- 172.21.15.115:0/2397002403 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559829336 0 0) 0x7f98b00c7440 con 0x7f98b015e8e0 2023-03-25T00:57:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b5e48700 1 --2- 172.21.15.115:0/2397002403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a620 0x7f98b00c6f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0098f0 tx=0x7f98ac009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f16f98b0e5687ff4 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b4e46700 1 -- 172.21.15.115:0/2397002403 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac011830 con 0x7f98b015a620 2023-03-25T00:57:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b4e46700 1 -- 172.21.15.115:0/2397002403 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac011990 con 0x7f98b015a620 2023-03-25T00:57:36.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.824+0000 7f98b4e46700 1 -- 172.21.15.115:0/2397002403 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac011d20 con 0x7f98b015a620 2023-03-25T00:57:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.825+0000 7f98b80ac700 1 -- 172.21.15.115:0/2397002403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a620 msgr2=0x7f98b00c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.825+0000 7f98b80ac700 1 --2- 172.21.15.115:0/2397002403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a620 0x7f98b00c6f00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f98ac0098f0 tx=0x7f98ac009c00 comp rx=0 tx=0).stop 2023-03-25T00:57:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.825+0000 7f98b80ac700 1 -- 172.21.15.115:0/2397002403 shutdown_connections 2023-03-25T00:57:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.825+0000 7f98b80ac700 1 --2- 172.21.15.115:0/2397002403 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a620 0x7f98b00c6f00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.825+0000 7f98b80ac700 1 --2- 172.21.15.115:0/2397002403 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015a9f0 0x7f98b015e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.825+0000 7f98b80ac700 1 -- 172.21.15.115:0/2397002403 >> 172.21.15.115:0/2397002403 conn(0x7f98b00d0b50 msgr2=0x7f98b00cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.825+0000 7f98b80ac700 1 -- 172.21.15.115:0/2397002403 shutdown_connections 2023-03-25T00:57:36.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.825+0000 7f98b80ac700 1 -- 172.21.15.115:0/2397002403 wait complete. 2023-03-25T00:57:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.827+0000 7f98b80ac700 1 Processor -- start 2023-03-25T00:57:36.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.827+0000 7f98b80ac700 1 -- start start 2023-03-25T00:57:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b80ac700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b015a620 0x7f98b014e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b80ac700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a9f0 0x7f98b014f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b80ac700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015e8e0 0x7f98b00b3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b80ac700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b01f1130 con 0x7f98b015a620 2023-03-25T00:57:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b80ac700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b0150640 con 0x7f98b015a9f0 2023-03-25T00:57:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b80ac700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b014fda0 con 0x7f98b015e8e0 2023-03-25T00:57:36.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b5e48700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b015a620 0x7f98b014e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b5e48700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b015a620 0x7f98b014e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32856/0 (socket says 172.21.15.115:32856) 2023-03-25T00:57:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b5e48700 1 -- 172.21.15.115:0/3242428086 learned_addr learned my addr 172.21.15.115:0/3242428086 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b5647700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a9f0 0x7f98b014f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.828+0000 7f98b6649700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015e8e0 0x7f98b00b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b5e48700 1 -- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015e8e0 msgr2=0x7f98b00b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b5e48700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015e8e0 0x7f98b00b3920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.834 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b5e48700 1 -- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a9f0 msgr2=0x7f98b014f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b5e48700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a9f0 0x7f98b014f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b5e48700 1 -- 172.21.15.115:0/3242428086 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98ac009550 con 0x7f98b015a620 2023-03-25T00:57:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b5647700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a9f0 0x7f98b014f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b5e48700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b015a620 0x7f98b014e4d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009c40 tx=0x7f98ac004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98a67fc700 1 -- 172.21.15.115:0/3242428086 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac004300 con 0x7f98b015a620 2023-03-25T00:57:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98a67fc700 1 -- 172.21.15.115:0/3242428086 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ac01f070 con 0x7f98b015a620 2023-03-25T00:57:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b00ca760 con 0x7f98b015a620 2023-03-25T00:57:36.835 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98a67fc700 1 -- 172.21.15.115:0/3242428086 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ac0036a0 con 0x7f98b015a620 2023-03-25T00:57:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.829+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b00ca480 con 0x7f98b015a620 2023-03-25T00:57:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.830+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b00caa40 con 0x7f98b015a620 2023-03-25T00:57:36.836 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.830+0000 7f98a67fc700 1 -- 172.21.15.115:0/3242428086 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7f98ac021030 con 0x7f98b015a620 2023-03-25T00:57:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.831+0000 7f98a67fc700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f9894069d50 0x7f989406c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.831+0000 7f98a67fc700 1 -- 172.21.15.115:0/3242428086 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f98ac088ee0 con 0x7f98b015a620 2023-03-25T00:57:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.831+0000 7f98b5647700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f9894069d50 0x7f989406c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:36.837 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.832+0000 7f98b5647700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f9894069d50 0x7f989406c200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f98b00c6cd0 tx=0x7f98a0009210 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:36.838 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:36.836+0000 7f98a67fc700 1 -- 172.21.15.115:0/3242428086 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f98ac054bd0 con 0x7f98b015a620 2023-03-25T00:57:37.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.031+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f98b0074580 con 0x7f98b015a620 2023-03-25T00:57:37.033 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.031+0000 7f98a67fc700 1 -- 172.21.15.115:0/3242428086 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v15) v1 ==== 92+0+10 (secure 0 0 0) 0x7f98ac0150a0 con 0x7f98b015a620 2023-03-25T00:57:37.033 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 8 2023-03-25T00:57:37.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f9894069d50 msgr2=0x7f989406c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.035 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f9894069d50 0x7f989406c200 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f98b00c6cd0 tx=0x7f98a0009210 comp rx=0 tx=0).stop 2023-03-25T00:57:37.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b015a620 msgr2=0x7f98b014e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b015a620 0x7f98b014e4d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f98ac009c40 tx=0x7f98ac004750 comp rx=0 tx=0).stop 2023-03-25T00:57:37.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 shutdown_connections 2023-03-25T00:57:37.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f9894069d50 0x7f989406c200 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f98b015e8e0 0x7f98b00b3920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f98b015a620 0x7f98b014e4d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.036 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 --2- 172.21.15.115:0/3242428086 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98b015a9f0 0x7f98b014f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 >> 172.21.15.115:0/3242428086 conn(0x7f98b00d0b50 msgr2=0x7f98b00715c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:37.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 shutdown_connections 2023-03-25T00:57:37.037 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.033+0000 7f98b80ac700 1 -- 172.21.15.115:0/3242428086 wait complete. 2023-03-25T00:57:37.047 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-25T00:57:37.169 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.167+0000 7f5c1e515700 1 Processor -- start 2023-03-25T00:57:37.170 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.167+0000 7f5c1e515700 1 -- start start 2023-03-25T00:57:37.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.168+0000 7f5c1e515700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c18070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.168+0000 7f5c1e515700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c18160610 con 0x7f5c18071520 2023-03-25T00:57:37.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.168+0000 7f5c1e515700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c181608f0 con 0x7f5c180cf7e0 2023-03-25T00:57:37.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.168+0000 7f5c1e515700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c180cf170 con 0x7f5c180ced00 2023-03-25T00:57:37.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.168+0000 7f5c1cab2700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f5c18071520 0x7f5c1815d370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33436/0 (socket says 172.21.15.115:33436) 2023-03-25T00:57:37.172 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.168+0000 7f5c1cab2700 1 -- 172.21.15.115:0/3691842955 learned_addr learned my addr 172.21.15.115:0/3691842955 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:37.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.168+0000 7f5c177fe700 1 --2- 172.21.15.115:0/3691842955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c18070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c16ffd700 1 -- 172.21.15.115:0/3691842955 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896641481 0 0) 0x7f5c18160610 con 0x7f5c18071520 2023-03-25T00:57:37.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c16ffd700 1 -- 172.21.15.115:0/3691842955 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c000035b0 con 0x7f5c18071520 2023-03-25T00:57:37.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c16ffd700 1 -- 172.21.15.115:0/3691842955 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1948163017 0 0) 0x7f5c181608f0 con 0x7f5c180cf7e0 2023-03-25T00:57:37.173 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c16ffd700 1 -- 172.21.15.115:0/3691842955 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c18160610 con 0x7f5c180cf7e0 2023-03-25T00:57:37.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c177fe700 1 -- 172.21.15.115:0/3691842955 >> v1:172.21.15.121:6789/0 conn(0x7f5c180cf7e0 legacy=0x7f5c180cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c177fe700 1 -- 172.21.15.115:0/3691842955 >> v1:172.21.15.115:6789/0 conn(0x7f5c18071520 legacy=0x7f5c1815d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c177fe700 1 -- 172.21.15.115:0/3691842955 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c18160bd0 con 0x7f5c180ced00 2023-03-25T00:57:37.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c177fe700 1 --2- 172.21.15.115:0/3691842955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c18070fe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009870 tx=0x7f5c08009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e290b9ca0cebcced server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c16ffd700 1 -- 172.21.15.115:0/3691842955 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08011830 con 0x7f5c180ced00 2023-03-25T00:57:37.174 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c16ffd700 1 -- 172.21.15.115:0/3691842955 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c08011990 con 0x7f5c180ced00 2023-03-25T00:57:37.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.169+0000 7f5c16ffd700 1 -- 172.21.15.115:0/3691842955 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c08011d20 con 0x7f5c180ced00 2023-03-25T00:57:37.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.170+0000 7f5c1e515700 1 -- 172.21.15.115:0/3691842955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 msgr2=0x7f5c18070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.170+0000 7f5c1e515700 1 --2- 172.21.15.115:0/3691842955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c18070fe0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009870 tx=0x7f5c08009b80 comp rx=0 tx=0).stop 2023-03-25T00:57:37.175 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.170+0000 7f5c1e515700 1 -- 172.21.15.115:0/3691842955 shutdown_connections 2023-03-25T00:57:37.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.170+0000 7f5c1e515700 1 --2- 172.21.15.115:0/3691842955 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c18070fe0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.170+0000 7f5c1e515700 1 -- 172.21.15.115:0/3691842955 >> 172.21.15.115:0/3691842955 conn(0x7f5c180d0b50 msgr2=0x7f5c180cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:37.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.170+0000 7f5c1e515700 1 -- 172.21.15.115:0/3691842955 shutdown_connections 2023-03-25T00:57:37.176 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.170+0000 7f5c1e515700 1 -- 172.21.15.115:0/3691842955 wait complete. 2023-03-25T00:57:37.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.171+0000 7f5c1e515700 1 Processor -- start 2023-03-25T00:57:37.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.171+0000 7f5c1e515700 1 -- start start 2023-03-25T00:57:37.179 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c1e515700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c18071520 0x7f5c181ef9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c1e515700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c181f2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c1e515700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c180cf7e0 0x7f5c181523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c1e515700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c181eff20 con 0x7f5c180cf7e0 2023-03-25T00:57:37.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c1e515700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c181f34a0 con 0x7f5c18071520 2023-03-25T00:57:37.180 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c1e515700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c18151a50 con 0x7f5c180ced00 2023-03-25T00:57:37.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c177fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c181f2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c17fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c18071520 0x7f5c181ef9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c177fe700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c181f2f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.121:3301/0 says I am v2:172.21.15.115:45220/0 (socket says 172.21.15.115:45220) 2023-03-25T00:57:37.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c177fe700 1 -- 172.21.15.115:0/514156104 learned_addr learned my addr 172.21.15.115:0/514156104 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:37.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c1cab2700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c180cf7e0 0x7f5c181523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.181 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c177fe700 1 -- 172.21.15.115:0/514156104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c18071520 msgr2=0x7f5c181ef9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c177fe700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c18071520 0x7f5c181ef9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c177fe700 1 -- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c180cf7e0 msgr2=0x7f5c181523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c177fe700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c180cf7e0 0x7f5c181523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.172+0000 7f5c177fe700 1 -- 172.21.15.115:0/514156104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c080094a0 con 0x7f5c180ced00 2023-03-25T00:57:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.173+0000 7f5c17fff700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c18071520 0x7f5c181ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.173+0000 7f5c1cab2700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c180cf7e0 0x7f5c181523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-25T00:57:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.173+0000 7f5c177fe700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c181f2f60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5c080090b0 tx=0x7f5c08004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.182 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.173+0000 7f5c14ff9700 1 -- 172.21.15.115:0/514156104 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0801a760 con 0x7f5c180ced00 2023-03-25T00:57:37.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.173+0000 7f5c14ff9700 1 -- 172.21.15.115:0/514156104 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c0801a8c0 con 0x7f5c180ced00 2023-03-25T00:57:37.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.173+0000 7f5c14ff9700 1 -- 172.21.15.115:0/514156104 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c0801ac50 con 0x7f5c180ced00 2023-03-25T00:57:37.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.173+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c18161190 con 0x7f5c180ced00 2023-03-25T00:57:37.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.173+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c18160eb0 con 0x7f5c180ced00 2023-03-25T00:57:37.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.174+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c18161470 con 0x7f5c180ced00 2023-03-25T00:57:37.183 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.174+0000 7f5c14ff9700 1 -- 172.21.15.115:0/514156104 <== mon.2 v2:172.21.15.121:3301/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7f5c08022070 con 0x7f5c180ced00 2023-03-25T00:57:37.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.174+0000 7f5c14ff9700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5c04069d50 0x7f5c0406c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.175+0000 7f5c14ff9700 1 -- 172.21.15.115:0/514156104 <== mon.2 v2:172.21.15.121:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f5c08088840 con 0x7f5c180ced00 2023-03-25T00:57:37.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.175+0000 7f5c17fff700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5c04069d50 0x7f5c0406c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.184 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.175+0000 7f5c17fff700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5c04069d50 0x7f5c0406c200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0011e0 tx=0x7f5c0c009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.185 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.179+0000 7f5c14ff9700 1 -- 172.21.15.115:0/514156104 <== mon.2 v2:172.21.15.121:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f5c080545b0 con 0x7f5c180ced00 2023-03-25T00:57:37.378 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.376+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5c180ceae0 con 0x7f5c180ced00 2023-03-25T00:57:37.389 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.388+0000 7f5c14ff9700 1 -- 172.21.15.115:0/514156104 <== mon.2 v2:172.21.15.121:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5c08029020 con 0x7f5c180ced00 2023-03-25T00:57:37.390 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-03-25T00:57:37.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5c04069d50 msgr2=0x7f5c0406c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5c04069d50 0x7f5c0406c200 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0011e0 tx=0x7f5c0c009040 comp rx=0 tx=0).stop 2023-03-25T00:57:37.393 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 msgr2=0x7f5c181f2f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c181f2f60 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5c080090b0 tx=0x7f5c08004750 comp rx=0 tx=0).stop 2023-03-25T00:57:37.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 shutdown_connections 2023-03-25T00:57:37.394 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5c04069d50 0x7f5c0406c200 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5c18071520 0x7f5c181ef9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5c180cf7e0 0x7f5c181523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.395 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 --2- 172.21.15.115:0/514156104 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f5c180ced00 0x7f5c181f2f60 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 >> 172.21.15.115:0/514156104 conn(0x7f5c180d0b50 msgr2=0x7f5c180ce250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:37.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 shutdown_connections 2023-03-25T00:57:37.396 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.390+0000 7f5c1e515700 1 -- 172.21.15.115:0/514156104 wait complete. 2023-03-25T00:57:37.405 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-25T00:57:37.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.523+0000 7fc8a144a700 1 Processor -- start 2023-03-25T00:57:37.525 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.523+0000 7fc8a144a700 1 -- start start 2023-03-25T00:57:37.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc8a144a700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc89c15d340 0x7fc89c0cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc8a144a700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc89c0cd800 0x7fc89c0cdc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc8a144a700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89c0cd370 con 0x7fc89c0b10f0 2023-03-25T00:57:37.527 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc8a144a700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89c0cf280 con 0x7fc89c15d340 2023-03-25T00:57:37.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc89affd700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc89c0b10f0 0x7fc89c0b14c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33454/0 (socket says 172.21.15.115:33454) 2023-03-25T00:57:37.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc8a144a700 1 -- 172.21.15.115:0/4125092566 --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89c15d850 con 0x7fc89c0cd800 2023-03-25T00:57:37.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc89affd700 1 -- 172.21.15.115:0/4125092566 learned_addr learned my addr 172.21.15.115:0/4125092566 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:37.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc89a7fc700 1 --2- 172.21.15.115:0/4125092566 >> v2:172.21.15.115:3300/0 conn(0x7fc89c15d340 0x7fc89c0cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.524+0000 7fc89b7fe700 1 --2- 172.21.15.115:0/4125092566 >> v2:172.21.15.121:3300/0 conn(0x7fc89c0cd800 0x7fc89c0cdc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc899ffb700 1 -- 172.21.15.115:0/4125092566 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3743561864 0 0) 0x7fc89c0cd370 con 0x7fc89c0b10f0 2023-03-25T00:57:37.528 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc899ffb700 1 -- 172.21.15.115:0/4125092566 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8880035b0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc89b7fe700 1 -- 172.21.15.115:0/4125092566 >> v2:172.21.15.115:3300/0 conn(0x7fc89c15d340 msgr2=0x7fc89c0cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:37.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc89b7fe700 1 --2- 172.21.15.115:0/4125092566 >> v2:172.21.15.115:3300/0 conn(0x7fc89c15d340 0x7fc89c0cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc89b7fe700 1 -- 172.21.15.115:0/4125092566 >> v1:172.21.15.115:6789/0 conn(0x7fc89c0b10f0 legacy=0x7fc89c0b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc89b7fe700 1 -- 172.21.15.115:0/4125092566 --> v2:172.21.15.121:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89c1591f0 con 0x7fc89c0cd800 2023-03-25T00:57:37.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc89a7fc700 1 --2- 172.21.15.115:0/4125092566 >> v2:172.21.15.115:3300/0 conn(0x7fc89c15d340 0x7fc89c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:37.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc89b7fe700 1 --2- 172.21.15.115:0/4125092566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 0x7fc89c0cdc60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc89c070620 tx=0x7fc88c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f6d6d2cd48160f6 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.529 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc899ffb700 1 -- 172.21.15.115:0/4125092566 <== mon.1 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c00e070 con 0x7fc89c0cd800 2023-03-25T00:57:37.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc899ffb700 1 -- 172.21.15.115:0/4125092566 <== mon.1 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88c00bb20 con 0x7fc89c0cd800 2023-03-25T00:57:37.530 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.525+0000 7fc899ffb700 1 -- 172.21.15.115:0/4125092566 <== mon.1 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88c012430 con 0x7fc89c0cd800 2023-03-25T00:57:37.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.526+0000 7fc8a144a700 1 -- 172.21.15.115:0/4125092566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 msgr2=0x7fc89c0cdc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.526+0000 7fc8a144a700 1 --2- 172.21.15.115:0/4125092566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 0x7fc89c0cdc60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc89c070620 tx=0x7fc88c00b330 comp rx=0 tx=0).stop 2023-03-25T00:57:37.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.526+0000 7fc8a144a700 1 -- 172.21.15.115:0/4125092566 shutdown_connections 2023-03-25T00:57:37.532 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.526+0000 7fc8a144a700 1 --2- 172.21.15.115:0/4125092566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 0x7fc89c0cdc60 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.526+0000 7fc8a144a700 1 --2- 172.21.15.115:0/4125092566 >> v2:172.21.15.115:3300/0 conn(0x7fc89c15d340 0x7fc89c0cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.526+0000 7fc8a144a700 1 -- 172.21.15.115:0/4125092566 >> 172.21.15.115:0/4125092566 conn(0x7fc89c06d880 msgr2=0x7fc89c0cc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:37.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.526+0000 7fc8a144a700 1 -- 172.21.15.115:0/4125092566 shutdown_connections 2023-03-25T00:57:37.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.526+0000 7fc8a144a700 1 -- 172.21.15.115:0/4125092566 wait complete. 2023-03-25T00:57:37.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.527+0000 7fc8a144a700 1 Processor -- start 2023-03-25T00:57:37.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.527+0000 7fc8a144a700 1 -- start start 2023-03-25T00:57:37.533 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8a144a700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc89c0b10f0 0x7fc89c0b35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8a144a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 0x7fc89c0c4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8a144a700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc89c15d340 0x7fc89c0c31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8a144a700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c0b3af0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8a144a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c15b240 con 0x7fc89c0cd800 2023-03-25T00:57:37.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8a144a700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc89c0c5bc0 con 0x7fc89c15d340 2023-03-25T00:57:37.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc89c0b10f0 0x7fc89c0b35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc89c0b10f0 0x7fc89c0b35b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32900/0 (socket says 172.21.15.115:32900) 2023-03-25T00:57:37.534 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 -- 172.21.15.115:0/2898034564 learned_addr learned my addr 172.21.15.115:0/2898034564 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:37.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 -- 172.21.15.115:0/2898034564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc89c15d340 msgr2=0x7fc89c0c31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc89c15d340 0x7fc89c0c31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 -- 172.21.15.115:0/2898034564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 msgr2=0x7fc89c0c4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89a7fc700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 0x7fc89c0c4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 0x7fc89c0c4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 -- 172.21.15.115:0/2898034564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8840094a0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.535 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89a7fc700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 0x7fc89c0c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc89affd700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc89c0b10f0 0x7fc89c0b35b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc894001eb0 tx=0x7fc89400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8937fe700 1 -- 172.21.15.115:0/2898034564 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc894005d70 con 0x7fc89c0b10f0 2023-03-25T00:57:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8937fe700 1 -- 172.21.15.115:0/2898034564 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8940070d0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c009a70 con 0x7fc89c0b10f0 2023-03-25T00:57:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8937fe700 1 -- 172.21.15.115:0/2898034564 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc894007460 con 0x7fc89c0b10f0 2023-03-25T00:57:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.528+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc89c0ca0d0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.529+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc89c0c9df0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.536 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.530+0000 7fc8937fe700 1 -- 172.21.15.115:0/2898034564 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7fc8940075c0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.530+0000 7fc8937fe700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc87c069d50 0x7fc87c06c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.530+0000 7fc89a7fc700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc87c069d50 0x7fc87c06c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.530+0000 7fc8937fe700 1 -- 172.21.15.115:0/2898034564 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc894087120 con 0x7fc89c0b10f0 2023-03-25T00:57:37.537 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.531+0000 7fc89a7fc700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc87c069d50 0x7fc87c06c200 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc89c1f1760 tx=0x7fc884008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.538 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.535+0000 7fc8937fe700 1 -- 172.21.15.115:0/2898034564 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7fc894052d70 con 0x7fc89c0b10f0 2023-03-25T00:57:37.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.770+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc89c0cbea0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.772 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.770+0000 7fc8937fe700 1 -- 172.21.15.115:0/2898034564 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3536 (secure 0 0 0) 0x7fc894017de0 con 0x7fc89c0b10f0 2023-03-25T00:57:37.773 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:57:37.773 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4217,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6837/1444364515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6835","nonce":1444364515},{"type":"v1","addr":"172.21.15.115:6837","nonce":1444364515}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4226,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6834/2044658443","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6832","nonce":2044658443},{"type":"v1","addr":"172.21.15.121:6834","nonce":2044658443}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4248,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.115:6836/2829118800","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6834","nonce":2829118800},{"type":"v1","addr":"172.21.15.115:6836","nonce":2829118800}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4266,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.121:6835/1975929757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6833","nonce":1975929757},{"type":"v1","addr":"172.21.15.121:6835","nonce":1975929757}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-03-25T00:57:37.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc87c069d50 msgr2=0x7fc87c06c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc87c069d50 0x7fc87c06c200 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc89c1f1760 tx=0x7fc884008040 comp rx=0 tx=0).stop 2023-03-25T00:57:37.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc89c0b10f0 msgr2=0x7fc89c0b35b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc89c0b10f0 0x7fc89c0b35b0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc894001eb0 tx=0x7fc89400c860 comp rx=0 tx=0).stop 2023-03-25T00:57:37.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 shutdown_connections 2023-03-25T00:57:37.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc87c069d50 0x7fc87c06c200 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.775 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc89c15d340 0x7fc89c0c31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc89c0b10f0 0x7fc89c0b35b0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 --2- 172.21.15.115:0/2898034564 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc89c0cd800 0x7fc89c0c4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 >> 172.21.15.115:0/2898034564 conn(0x7fc89c06d880 msgr2=0x7fc89c0af430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:37.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 shutdown_connections 2023-03-25T00:57:37.776 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.772+0000 7fc8a144a700 1 -- 172.21.15.115:0/2898034564 wait complete. 2023-03-25T00:57:37.776 INFO:teuthology.orchestra.run.smithi115.stderr:dumped fsmap epoch 3 2023-03-25T00:57:37.787 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-25T00:57:37.906 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.904+0000 7f777bbf0700 1 Processor -- start 2023-03-25T00:57:37.907 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.905+0000 7f777bbf0700 1 -- start start 2023-03-25T00:57:37.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.905+0000 7f777bbf0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77740b1660 0x7f77740cc7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.908 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.905+0000 7f777bbf0700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77740cd7e0 con 0x7f77740cccf0 2023-03-25T00:57:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.905+0000 7f777bbf0700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77740cd220 con 0x7f777415a770 2023-03-25T00:57:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.905+0000 7f777bbf0700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77740cd500 con 0x7f77740b1660 2023-03-25T00:57:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777a18d700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f77740cccf0 0x7f77740cda30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33480/0 (socket says 172.21.15.115:33480) 2023-03-25T00:57:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777a18d700 1 -- 172.21.15.115:0/3441105900 learned_addr learned my addr 172.21.15.115:0/3441105900 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777918b700 1 --2- 172.21.15.115:0/3441105900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77740b1660 0x7f77740cc7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777898a700 1 -- 172.21.15.115:0/3441105900 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824699936 0 0) 0x7f77740cd7e0 con 0x7f77740cccf0 2023-03-25T00:57:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777898a700 1 -- 172.21.15.115:0/3441105900 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77600035b0 con 0x7f77740cccf0 2023-03-25T00:57:37.909 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777898a700 1 -- 172.21.15.115:0/3441105900 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320285169 0 0) 0x7f77740cd220 con 0x7f777415a770 2023-03-25T00:57:37.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777898a700 1 -- 172.21.15.115:0/3441105900 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77740cd7e0 con 0x7f777415a770 2023-03-25T00:57:37.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777918b700 1 -- 172.21.15.115:0/3441105900 >> v1:172.21.15.121:6789/0 conn(0x7f777415a770 legacy=0x7f77740b3e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777918b700 1 -- 172.21.15.115:0/3441105900 >> v1:172.21.15.115:6789/0 conn(0x7f77740cccf0 legacy=0x7f77740cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777918b700 1 -- 172.21.15.115:0/3441105900 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77740b27e0 con 0x7f77740b1660 2023-03-25T00:57:37.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777918b700 1 --2- 172.21.15.115:0/3441105900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77740b1660 0x7f77740cc7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7770009870 tx=0x7f7770009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6a563a0edccf814 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777898a700 1 -- 172.21.15.115:0/3441105900 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770011830 con 0x7f77740b1660 2023-03-25T00:57:37.910 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777898a700 1 -- 172.21.15.115:0/3441105900 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7770011990 con 0x7f77740b1660 2023-03-25T00:57:37.911 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.906+0000 7f777898a700 1 -- 172.21.15.115:0/3441105900 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7770011d20 con 0x7f77740b1660 2023-03-25T00:57:37.912 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.907+0000 7f777bbf0700 1 -- 172.21.15.115:0/3441105900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77740b1660 msgr2=0x7f77740cc7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.907+0000 7f777bbf0700 1 --2- 172.21.15.115:0/3441105900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77740b1660 0x7f77740cc7b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7770009870 tx=0x7f7770009b80 comp rx=0 tx=0).stop 2023-03-25T00:57:37.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.907+0000 7f777bbf0700 1 -- 172.21.15.115:0/3441105900 shutdown_connections 2023-03-25T00:57:37.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.907+0000 7f777bbf0700 1 --2- 172.21.15.115:0/3441105900 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f77740b1660 0x7f77740cc7b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.907+0000 7f777bbf0700 1 -- 172.21.15.115:0/3441105900 >> 172.21.15.115:0/3441105900 conn(0x7f7774070060 msgr2=0x7f7774070430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:37.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.907+0000 7f777bbf0700 1 -- 172.21.15.115:0/3441105900 shutdown_connections 2023-03-25T00:57:37.913 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.907+0000 7f777bbf0700 1 -- 172.21.15.115:0/3441105900 wait complete. 2023-03-25T00:57:37.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.908+0000 7f777bbf0700 1 Processor -- start 2023-03-25T00:57:37.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.908+0000 7f777bbf0700 1 -- start start 2023-03-25T00:57:37.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777bbf0700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f77740b1660 0x7f77741f0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777bbf0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77740cccf0 0x7f777415c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777bbf0700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777415a770 0x7f777414e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777bbf0700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777414ead0 con 0x7f77740b1660 2023-03-25T00:57:37.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777bbf0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777415e120 con 0x7f77740cccf0 2023-03-25T00:57:37.914 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777bbf0700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77740c8ea0 con 0x7f777415a770 2023-03-25T00:57:37.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f77740b1660 0x7f77741f0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f77740b1660 0x7f77741f0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32922/0 (socket says 172.21.15.115:32922) 2023-03-25T00:57:37.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 -- 172.21.15.115:0/4282528380 learned_addr learned my addr 172.21.15.115:0/4282528380 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:37.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777918b700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77740cccf0 0x7f777415c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777a18d700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777415a770 0x7f777414e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 -- 172.21.15.115:0/4282528380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777415a770 msgr2=0x7f777414e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777415a770 0x7f777414e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 -- 172.21.15.115:0/4282528380 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77740cccf0 msgr2=0x7f777415c4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:37.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77740cccf0 0x7f777415c4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:37.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 -- 172.21.15.115:0/4282528380 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77700094a0 con 0x7f77740b1660 2023-03-25T00:57:37.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777998c700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f77740b1660 0x7f77741f0170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f776400c530 tx=0x7f776400c840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f776a7fc700 1 -- 172.21.15.115:0/4282528380 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776400cea0 con 0x7f77740b1660 2023-03-25T00:57:37.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77740b4580 con 0x7f77740b1660 2023-03-25T00:57:37.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f776a7fc700 1 -- 172.21.15.115:0/4282528380 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77640052f0 con 0x7f77740b1660 2023-03-25T00:57:37.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.909+0000 7f776a7fc700 1 -- 172.21.15.115:0/4282528380 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764003680 con 0x7f77740b1660 2023-03-25T00:57:37.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.910+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77740b2ac0 con 0x7f77740b1660 2023-03-25T00:57:37.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.910+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777415ab40 con 0x7f77740b1660 2023-03-25T00:57:37.917 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.911+0000 7f776a7fc700 1 -- 172.21.15.115:0/4282528380 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7f77640037e0 con 0x7f77740b1660 2023-03-25T00:57:37.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.911+0000 7f776a7fc700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7758069d50 0x7f775806c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:37.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.911+0000 7f777918b700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7758069d50 0x7f775806c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:37.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.911+0000 7f776a7fc700 1 -- 172.21.15.115:0/4282528380 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f7764086eb0 con 0x7f77740b1660 2023-03-25T00:57:37.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.912+0000 7f777918b700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7758069d50 0x7f775806c200 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77740b1a30 tx=0x7f777001c040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:37.918 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.916+0000 7f776a7fc700 1 -- 172.21.15.115:0/4282528380 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f7764052ba0 con 0x7f77740b1660 2023-03-25T00:57:37.966 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:37.964+0000 7f776a7fc700 1 -- 172.21.15.115:0/4282528380 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f7764052f70 con 0x7f77740b1660 2023-03-25T00:57:38.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.108+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7774074580 con 0x7f77740b1660 2023-03-25T00:57:38.110 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.109+0000 7f776a7fc700 1 -- 172.21.15.115:0/4282528380 <== mon.0 v2:172.21.15.115:3300/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+14159 (secure 0 0 0) 0x7f7764017090 con 0x7f77740b1660 2023-03-25T00:57:38.111 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:57:38.111 INFO:teuthology.orchestra.run.smithi115.stdout:{"epoch":16,"fsid":"9c1ea6b6-7c28-4560-af27-3b583d106388","created":"2023-03-25T00:57:23.615145+0000","modified":"2023-03-25T00:57:37.291030+0000","last_up_change":"2023-03-25T00:57:34.273020+0000","last_in_change":"2023-03-25T00:57:26.100484+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":"rbd","create_time":"2023-03-25T00:57:34.908808+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":".mgr","create_time":"2023-03-25T00:57:35.207270+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b6e324a3-c8ad-459c-bb1e-256d32a317db","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6816","nonce":97654},{"type":"v1","addr":"172.21.15.115:6817","nonce":97654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6818","nonce":97654},{"type":"v1","addr":"172.21.15.115:6819","nonce":97654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6822","nonce":97654},{"type":"v1","addr":"172.21.15.115:6823","nonce":97654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6820","nonce":97654},{"type":"v1","addr":"172.21.15.115:6821","nonce":97654}]},"public_addr":"172.21.15.115:6817/97654","cluster_addr":"172.21.15.115:6819/97654","heartbeat_back_addr":"172.21.15.115:6823/97654","heartbeat_front_addr":"172.21.15.115:6821/97654","state":["exists","up"]},{"osd":1,"uuid":"009b7c98-0c48-49f2-98a1-e8d7afe9af58","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6808","nonce":97652},{"type":"v1","addr":"172.21.15.115:6809","nonce":97652}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6810","nonce":97652},{"type":"v1","addr":"172.21.15.115:6811","nonce":97652}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6814","nonce":97652},{"type":"v1","addr":"172.21.15.115:6815","nonce":97652}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6812","nonce":97652},{"type":"v1","addr":"172.21.15.115:6813","nonce":97652}]},"public_addr":"172.21.15.115:6809/97652","cluster_addr":"172.21.15.115:6811/97652","heartbeat_back_addr":"172.21.15.115:6815/97652","heartbeat_front_addr":"172.21.15.115:6813/97652","state":["exists","up"]},{"osd":2,"uuid":"b4056d77-ca36-481f-9779-017fb707bda5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6800","nonce":97653},{"type":"v1","addr":"172.21.15.115:6801","nonce":97653}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6802","nonce":97653},{"type":"v1","addr":"172.21.15.115:6803","nonce":97653}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6806","nonce":97653},{"type":"v1","addr":"172.21.15.115:6807","nonce":97653}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6804","nonce":97653},{"type":"v1","addr":"172.21.15.115:6805","nonce":97653}]},"public_addr":"172.21.15.115:6801/97653","cluster_addr":"172.21.15.115:6803/97653","heartbeat_back_addr":"172.21.15.115:6807/97653","heartbeat_front_addr":"172.21.15.115:6805/97653","state":["exists","up"]},{"osd":3,"uuid":"2c09166f-791d-42cf-9ef4-4746592f74e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6824","nonce":97666},{"type":"v1","addr":"172.21.15.115:6825","nonce":97666}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6826","nonce":97666},{"type":"v1","addr":"172.21.15.115:6827","nonce":97666}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6830","nonce":97666},{"type":"v1","addr":"172.21.15.115:6831","nonce":97666}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.115:6828","nonce":97666},{"type":"v1","addr":"172.21.15.115:6829","nonce":97666}]},"public_addr":"172.21.15.115:6825/97666","cluster_addr":"172.21.15.115:6827/97666","heartbeat_back_addr":"172.21.15.115:6831/97666","heartbeat_front_addr":"172.21.15.115:6829/97666","state":["exists","up"]},{"osd":4,"uuid":"2caea232-d588-4aac-bde0-39dae45860ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6816","nonce":97848},{"type":"v1","addr":"172.21.15.121:6817","nonce":97848}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6818","nonce":97848},{"type":"v1","addr":"172.21.15.121:6819","nonce":97848}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6822","nonce":97848},{"type":"v1","addr":"172.21.15.121:6823","nonce":97848}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6820","nonce":97848},{"type":"v1","addr":"172.21.15.121:6821","nonce":97848}]},"public_addr":"172.21.15.121:6817/97848","cluster_addr":"172.21.15.121:6819/97848","heartbeat_back_addr":"172.21.15.121:6823/97848","heartbeat_front_addr":"172.21.15.121:6821/97848","state":["exists","up"]},{"osd":5,"uuid":"cefc141f-68fb-4be4-957b-85ce0049ada5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6800","nonce":97844},{"type":"v1","addr":"172.21.15.121:6801","nonce":97844}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6802","nonce":97844},{"type":"v1","addr":"172.21.15.121:6803","nonce":97844}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6806","nonce":97844},{"type":"v1","addr":"172.21.15.121:6807","nonce":97844}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6804","nonce":97844},{"type":"v1","addr":"172.21.15.121:6805","nonce":97844}]},"public_addr":"172.21.15.121:6801/97844","cluster_addr":"172.21.15.121:6803/97844","heartbeat_back_addr":"172.21.15.121:6807/97844","heartbeat_front_addr":"172.21.15.121:6805/97844","state":["exists","up"]},{"osd":6,"uuid":"325c5ead-d0cd-45e2-b16c-d967f661d738","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6808","nonce":97845},{"type":"v1","addr":"172.21.15.121:6809","nonce":97845}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6810","nonce":97845},{"type":"v1","addr":"172.21.15.121:6811","nonce":97845}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6814","nonce":97845},{"type":"v1","addr":"172.21.15.121:6815","nonce":97845}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6812","nonce":97845},{"type":"v1","addr":"172.21.15.121:6813","nonce":97845}]},"public_addr":"172.21.15.121:6809/97845","cluster_addr":"172.21.15.121:6811/97845","heartbeat_back_addr":"172.21.15.121:6815/97845","heartbeat_front_addr":"172.21.15.121:6813/97845","state":["exists","up"]},{"osd":7,"uuid":"274ccf94-d180-4a8c-a0c3-f620f738252c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6824","nonce":97871},{"type":"v1","addr":"172.21.15.121:6825","nonce":97871}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6826","nonce":97871},{"type":"v1","addr":"172.21.15.121:6827","nonce":97871}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6830","nonce":97871},{"type":"v1","addr":"172.21.15.121:6831","nonce":97871}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:6828","nonce":97871},{"type":"v1","addr":"172.21.15.121:6829","nonce":97871}]},"public_addr":"172.21.15.121:6825/97871","cluster_addr":"172.21.15.121:6827/97871","heartbeat_back_addr":"172.21.15.121:6831/97871","heartbeat_front_addr":"172.21.15.121:6829/97871","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.246439+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.328118+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:30.993313+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.265418+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.261797+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.284125+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.233058+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-25T00:57:31.330665+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-25T00:57:38.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7758069d50 msgr2=0x7f775806c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7758069d50 0x7f775806c200 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f77740b1a30 tx=0x7f777001c040 comp rx=0 tx=0).stop 2023-03-25T00:57:38.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f77740b1660 msgr2=0x7f77741f0170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.113 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f77740b1660 0x7f77741f0170 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f776400c530 tx=0x7f776400c840 comp rx=0 tx=0).stop 2023-03-25T00:57:38.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 shutdown_connections 2023-03-25T00:57:38.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7758069d50 0x7f775806c200 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f777415a770 0x7f777414e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f77740b1660 0x7f77741f0170 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 --2- 172.21.15.115:0/4282528380 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77740cccf0 0x7f777415c4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 >> 172.21.15.115:0/4282528380 conn(0x7f7774070060 msgr2=0x7f77740b2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:38.114 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 shutdown_connections 2023-03-25T00:57:38.115 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.111+0000 7f777bbf0700 1 -- 172.21.15.115:0/4282528380 wait complete. 2023-03-25T00:57:38.125 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2023-03-25T00:57:34.908808+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': '.mgr', 'create_time': '2023-03-25T00:57:35.207270+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-25T00:57:38.125 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-25T00:57:38.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.246+0000 7fc0bceb8700 1 Processor -- start 2023-03-25T00:57:38.248 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.246+0000 7fc0bceb8700 1 -- start start 2023-03-25T00:57:38.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0bceb8700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc0b80cda00 0x7fc0b8070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0bceb8700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7fc0b815cfd0 0x7fc0b8071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0bceb8700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0b80ce900 con 0x7fc0b80cddd0 2023-03-25T00:57:38.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0bceb8700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b815e080 con 0x7fc0b80cda00 2023-03-25T00:57:38.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0bceb8700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0b80cfd10 con 0x7fc0b815cfd0 2023-03-25T00:57:38.251 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b659c700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7fc0b80cda00 0x7fc0b8070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b6d9d700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7fc0b80cddd0 0x7fc0b80ce1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33492/0 (socket says 172.21.15.115:33492) 2023-03-25T00:57:38.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b6d9d700 1 -- 172.21.15.115:0/3976990844 learned_addr learned my addr 172.21.15.115:0/3976990844 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:38.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b5d9b700 1 --2- 172.21.15.115:0/3976990844 >> v2:172.21.15.121:3300/0 conn(0x7fc0b815cfd0 0x7fc0b8071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b659c700 1 -- 172.21.15.115:0/3976990844 >> v2:172.21.15.121:3300/0 conn(0x7fc0b815cfd0 msgr2=0x7fc0b8071520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:38.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b659c700 1 --2- 172.21.15.115:0/3976990844 >> v2:172.21.15.121:3300/0 conn(0x7fc0b815cfd0 0x7fc0b8071520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b659c700 1 -- 172.21.15.115:0/3976990844 >> v1:172.21.15.115:6789/0 conn(0x7fc0b80cddd0 legacy=0x7fc0b80ce1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.252 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b659c700 1 -- 172.21.15.115:0/3976990844 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b815d470 con 0x7fc0b80cda00 2023-03-25T00:57:38.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b559a700 1 -- 172.21.15.115:0/3976990844 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740193443 0 0) 0x7fc0b80ce900 con 0x7fc0b80cddd0 2023-03-25T00:57:38.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.247+0000 7fc0b659c700 1 --2- 172.21.15.115:0/3976990844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cda00 0x7fc0b8070fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8072310 tx=0x7fc0ac009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbfa7ffc260aa45 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0b559a700 1 -- 172.21.15.115:0/3976990844 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac011830 con 0x7fc0b80cda00 2023-03-25T00:57:38.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0b559a700 1 -- 172.21.15.115:0/3976990844 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0ac011990 con 0x7fc0b80cda00 2023-03-25T00:57:38.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0b559a700 1 -- 172.21.15.115:0/3976990844 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac011d20 con 0x7fc0b80cda00 2023-03-25T00:57:38.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0bceb8700 1 -- 172.21.15.115:0/3976990844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cda00 msgr2=0x7fc0b8070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/3976990844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cda00 0x7fc0b8070fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8072310 tx=0x7fc0ac009b20 comp rx=0 tx=0).stop 2023-03-25T00:57:38.253 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0bceb8700 1 -- 172.21.15.115:0/3976990844 shutdown_connections 2023-03-25T00:57:38.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/3976990844 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cda00 0x7fc0b8070fe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/3976990844 >> v2:172.21.15.121:3300/0 conn(0x7fc0b815cfd0 0x7fc0b8071520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0bceb8700 1 -- 172.21.15.115:0/3976990844 >> 172.21.15.115:0/3976990844 conn(0x7fc0b80d0b50 msgr2=0x7fc0b80cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:38.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0bceb8700 1 -- 172.21.15.115:0/3976990844 shutdown_connections 2023-03-25T00:57:38.254 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.248+0000 7fc0bceb8700 1 -- 172.21.15.115:0/3976990844 wait complete. 2023-03-25T00:57:38.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0bceb8700 1 Processor -- start 2023-03-25T00:57:38.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0bceb8700 1 -- start start 2023-03-25T00:57:38.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0bceb8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b80cda00 0x7fc0b81f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0bceb8700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cddd0 0x7fc0b815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0bceb8700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0b815cfd0 0x7fc0b80c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0bceb8700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80c9200 con 0x7fc0b80cddd0 2023-03-25T00:57:38.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0bceb8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b81f3490 con 0x7fc0b80cda00 2023-03-25T00:57:38.257 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0bceb8700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0b80c4d30 con 0x7fc0b815cfd0 2023-03-25T00:57:38.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0b5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cddd0 0x7fc0b815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0b5d9b700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cddd0 0x7fc0b815bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32938/0 (socket says 172.21.15.115:32938) 2023-03-25T00:57:38.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.250+0000 7fc0b5d9b700 1 -- 172.21.15.115:0/53463414 learned_addr learned my addr 172.21.15.115:0/53463414 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:38.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b6d9d700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0b815cfd0 0x7fc0b80c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b659c700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b80cda00 0x7fc0b81f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b5d9b700 1 -- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0b815cfd0 msgr2=0x7fc0b80c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b5d9b700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0b815cfd0 0x7fc0b80c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b5d9b700 1 -- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b80cda00 msgr2=0x7fc0b81f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b5d9b700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b80cda00 0x7fc0b81f2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b5d9b700 1 -- 172.21.15.115:0/53463414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0ac0094a0 con 0x7fc0b80cddd0 2023-03-25T00:57:38.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b659c700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b80cda00 0x7fc0b81f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:38.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0b5d9b700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cddd0 0x7fc0b815bbc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800d800 tx=0x7fc0a800dbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0a6ffd700 1 -- 172.21.15.115:0/53463414 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a80125f0 con 0x7fc0b80cddd0 2023-03-25T00:57:38.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0a6ffd700 1 -- 172.21.15.115:0/53463414 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0a8012750 con 0x7fc0b80cddd0 2023-03-25T00:57:38.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b815da30 con 0x7fc0b80cddd0 2023-03-25T00:57:38.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0a6ffd700 1 -- 172.21.15.115:0/53463414 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0a8012b00 con 0x7fc0b80cddd0 2023-03-25T00:57:38.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.251+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0b815d750 con 0x7fc0b80cddd0 2023-03-25T00:57:38.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.252+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0b815dd10 con 0x7fc0b80cddd0 2023-03-25T00:57:38.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.252+0000 7fc0a6ffd700 1 -- 172.21.15.115:0/53463414 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7fc0a8012c60 con 0x7fc0b80cddd0 2023-03-25T00:57:38.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.253+0000 7fc0a6ffd700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc098069de0 0x7fc09806c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.253+0000 7fc0a6ffd700 1 -- 172.21.15.115:0/53463414 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fc0a8088100 con 0x7fc0b80cddd0 2023-03-25T00:57:38.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.253+0000 7fc0b659c700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc098069de0 0x7fc09806c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.253+0000 7fc0b659c700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc098069de0 0x7fc09806c290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80caf10 tx=0x7fc0ac008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.257+0000 7fc0a6ffd700 1 -- 172.21.15.115:0/53463414 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7fc0a8053c60 con 0x7fc0b80cddd0 2023-03-25T00:57:38.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.446+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc0b80ceae0 con 0x7fc0b80cddd0 2023-03-25T00:57:38.448 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.446+0000 7fc0a6ffd700 1 -- 172.21.15.115:0/53463414 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v16) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc0a8057280 con 0x7fc0b80cddd0 2023-03-25T00:57:38.448 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 8 2023-03-25T00:57:38.450 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc098069de0 msgr2=0x7fc09806c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc098069de0 0x7fc09806c290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0b80caf10 tx=0x7fc0ac008030 comp rx=0 tx=0).stop 2023-03-25T00:57:38.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cddd0 msgr2=0x7fc0b815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cddd0 0x7fc0b815bbc0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc0a800d800 tx=0x7fc0a800dbc0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 shutdown_connections 2023-03-25T00:57:38.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7fc098069de0 0x7fc09806c290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.451 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc0b80cda00 0x7fc0b81f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7fc0b815cfd0 0x7fc0b80c6f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 --2- 172.21.15.115:0/53463414 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7fc0b80cddd0 0x7fc0b815bbc0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 >> 172.21.15.115:0/53463414 conn(0x7fc0b80d0b50 msgr2=0x7fc0b80b0320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:38.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 shutdown_connections 2023-03-25T00:57:38.452 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.448+0000 7fc0bceb8700 1 -- 172.21.15.115:0/53463414 wait complete. 2023-03-25T00:57:38.462 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-25T00:57:38.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.582+0000 7f1db63a5700 1 Processor -- start 2023-03-25T00:57:38.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.582+0000 7f1db63a5700 1 -- start start 2023-03-25T00:57:38.592 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1db63a5700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1db0159940 0x7f1db00b4ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1db63a5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db00cb960 0x7f1db00cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1db63a5700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1db00cb170 con 0x7f1db00b5000 2023-03-25T00:57:38.593 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1db63a5700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1db00cc320 con 0x7f1db0159940 2023-03-25T00:57:38.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1db63a5700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00cb450 con 0x7f1db00cb960 2023-03-25T00:57:38.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1daf7fe700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f1db00b5000 0x7f1db00cda00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33504/0 (socket says 172.21.15.115:33504) 2023-03-25T00:57:38.594 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1daf7fe700 1 -- 172.21.15.115:0/3005789472 learned_addr learned my addr 172.21.15.115:0/3005789472 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1db4942700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db00cb960 0x7f1db00cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1daffff700 1 --2- 172.21.15.115:0/3005789472 >> v2:172.21.15.121:3300/0 conn(0x7f1db0159940 0x7f1db00b4ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.583+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922819617 0 0) 0x7f1db00cb170 con 0x7f1db00b5000 2023-03-25T00:57:38.595 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d980035b0 con 0x7f1db00b5000 2023-03-25T00:57:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4027328425 0 0) 0x7f1d980035b0 con 0x7f1db00b5000 2023-03-25T00:57:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db00cb960 msgr2=0x7f1db00cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db00cb960 0x7f1db00cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.596 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 >> v2:172.21.15.121:3300/0 conn(0x7f1db0159940 msgr2=0x7f1db00b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 --2- 172.21.15.115:0/3005789472 >> v2:172.21.15.121:3300/0 conn(0x7f1db0159940 0x7f1db00b4ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db00b42c0 con 0x7f1db00b5000 2023-03-25T00:57:38.597 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2816979759 0 0) 0x7f1da0001670 con 0x7f1db00b5000 2023-03-25T00:57:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 >> v1:172.21.15.115:6789/0 conn(0x7f1db00b5000 legacy=0x7f1db00cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1d98003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d98004430 0x7f1d980048f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.598 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db00cb960 0x7f1d98004fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00cc320 con 0x7f1d98003a00 2023-03-25T00:57:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00cb450 con 0x7f1d98004430 2023-03-25T00:57:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d9800ba10 con 0x7f1db00cb960 2023-03-25T00:57:38.599 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daffff700 1 --2- 172.21.15.115:0/3005789472 >> v2:172.21.15.121:3300/0 conn(0x7f1db0159940 0x7f1db00b4ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daffff700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1d98003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daf7fe700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d98004430 0x7f1d980048f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.600 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1db4942700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db00cb960 0x7f1d98004fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daffff700 1 -- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db00cb960 msgr2=0x7f1d98004fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daffff700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db00cb960 0x7f1d98004fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daffff700 1 -- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d98004430 msgr2=0x7f1d980048f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daffff700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1d98004430 0x7f1d980048f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daffff700 1 -- 172.21.15.115:0/3005789472 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d980035b0 con 0x7f1d98003a00 2023-03-25T00:57:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daffff700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1d98003e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009d10 tx=0x7f1d98005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9175d5c158a6f5a server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.601 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4009e60 con 0x7f1d98003a00 2023-03-25T00:57:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da400e9a0 con 0x7f1d98003a00 2023-03-25T00:57:38.602 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.584+0000 7f1daeffd700 1 -- 172.21.15.115:0/3005789472 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da400ed10 con 0x7f1d98003a00 2023-03-25T00:57:38.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.585+0000 7f1db63a5700 1 -- 172.21.15.115:0/3005789472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 msgr2=0x7f1d98003e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.585+0000 7f1db63a5700 1 --2- 172.21.15.115:0/3005789472 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1d98003e10 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f1da4009d10 tx=0x7f1d98005880 comp rx=0 tx=0).stop 2023-03-25T00:57:38.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.585+0000 7f1db4942700 1 -- 172.21.15.115:0/3005789472 reap_dead start 2023-03-25T00:57:38.603 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.585+0000 7f1db63a5700 1 -- 172.21.15.115:0/3005789472 shutdown_connections 2023-03-25T00:57:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.585+0000 7f1db63a5700 1 -- 172.21.15.115:0/3005789472 >> 172.21.15.115:0/3005789472 conn(0x7f1db00d0b50 msgr2=0x7f1db00cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.585+0000 7f1db63a5700 1 -- 172.21.15.115:0/3005789472 shutdown_connections 2023-03-25T00:57:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.585+0000 7f1db63a5700 1 -- 172.21.15.115:0/3005789472 wait complete. 2023-03-25T00:57:38.604 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.587+0000 7f1db63a5700 1 Processor -- start 2023-03-25T00:57:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.587+0000 7f1db63a5700 1 -- start start 2023-03-25T00:57:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1db63a5700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1db015a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1db63a5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1db014ae50 0x7f1db015c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.605 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1db63a5700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db006e1b0 0x7f1db00c9560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1db63a5700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00c9140 con 0x7f1d98003a00 2023-03-25T00:57:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1db63a5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00c8b70 con 0x7f1db014ae50 2023-03-25T00:57:38.606 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1db63a5700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1db00c8da0 con 0x7f1db006e1b0 2023-03-25T00:57:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1db015a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1db015a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32954/0 (socket says 172.21.15.115:32954) 2023-03-25T00:57:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 -- 172.21.15.115:0/1959257565 learned_addr learned my addr 172.21.15.115:0/1959257565 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:38.607 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daf7fe700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1db014ae50 0x7f1db015c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 -- 172.21.15.115:0/1959257565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db006e1b0 msgr2=0x7f1db00c9560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db006e1b0 0x7f1db00c9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 -- 172.21.15.115:0/1959257565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1db014ae50 msgr2=0x7f1db015c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.608 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1db014ae50 0x7f1db015c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 -- 172.21.15.115:0/1959257565 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db00b3f00 con 0x7f1d98003a00 2023-03-25T00:57:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daf7fe700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1db014ae50 0x7f1db015c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1daffff700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1db015a170 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1da40090b0 tx=0x7f1da40047c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1dacff9700 1 -- 172.21.15.115:0/1959257565 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4021070 con 0x7f1d98003a00 2023-03-25T00:57:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1dacff9700 1 -- 172.21.15.115:0/1959257565 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1da4009e60 con 0x7f1d98003a00 2023-03-25T00:57:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db00b45a0 con 0x7f1d98003a00 2023-03-25T00:57:38.609 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1dacff9700 1 -- 172.21.15.115:0/1959257565 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1da4003ad0 con 0x7f1d98003a00 2023-03-25T00:57:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.588+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db00b4090 con 0x7f1d98003a00 2023-03-25T00:57:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.589+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db00c70a0 con 0x7f1d98003a00 2023-03-25T00:57:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.590+0000 7f1dacff9700 1 -- 172.21.15.115:0/1959257565 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f1da4003c30 con 0x7f1d98003a00 2023-03-25T00:57:38.610 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.590+0000 7f1dacff9700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1d9c069e10 0x7f1d9c06c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.590+0000 7f1daf7fe700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1d9c069e10 0x7f1d9c06c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.590+0000 7f1dacff9700 1 -- 172.21.15.115:0/1959257565 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1da4088c60 con 0x7f1d98003a00 2023-03-25T00:57:38.611 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.590+0000 7f1daf7fe700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1d9c069e10 0x7f1d9c06c2c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1da0000fb0 tx=0x7f1da0009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.612 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.595+0000 7f1dacff9700 1 -- 172.21.15.115:0/1959257565 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f1da40548f0 con 0x7f1d98003a00 2023-03-25T00:57:38.781 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.779+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1db00c5ef0 con 0x7f1d98003a00 2023-03-25T00:57:38.782 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.780+0000 7f1dacff9700 1 -- 172.21.15.115:0/1959257565 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1da4057f10 con 0x7f1d98003a00 2023-03-25T00:57:38.782 INFO:teuthology.orchestra.run.smithi115.stdout:pg_num: 1 2023-03-25T00:57:38.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1d9c069e10 msgr2=0x7f1d9c06c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1d9c069e10 0x7f1d9c06c2c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1da0000fb0 tx=0x7f1da0009040 comp rx=0 tx=0).stop 2023-03-25T00:57:38.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 msgr2=0x7f1db015a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1db015a170 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f1da40090b0 tx=0x7f1da40047c0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.784 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 shutdown_connections 2023-03-25T00:57:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1d9c069e10 0x7f1d9c06c2c0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f1db006e1b0 0x7f1db00c9560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1d98003a00 0x7f1db015a170 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 --2- 172.21.15.115:0/1959257565 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1db014ae50 0x7f1db015c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 >> 172.21.15.115:0/1959257565 conn(0x7f1db00d0b50 msgr2=0x7f1db00c5590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 shutdown_connections 2023-03-25T00:57:38.785 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.782+0000 7f1db63a5700 1 -- 172.21.15.115:0/1959257565 wait complete. 2023-03-25T00:57:38.796 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-25T00:57:38.915 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.913+0000 7f71538d3700 1 Processor -- start 2023-03-25T00:57:38.916 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.913+0000 7f71538d3700 1 -- start start 2023-03-25T00:57:38.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f71538d3700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f714c0ce1d0 0x7f714c167870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f71538d3700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c0cf140 con 0x7f714c1602c0 2023-03-25T00:57:38.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f71538d3700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f714c0ce6a0 con 0x7f714c071760 2023-03-25T00:57:38.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f71538d3700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f714c0b4590 con 0x7f714c0ce1d0 2023-03-25T00:57:38.919 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f7151e70700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f714c0ce1d0 0x7f714c167870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f7151e70700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f714c0ce1d0 0x7f714c167870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32964/0 (socket says 172.21.15.115:32964) 2023-03-25T00:57:38.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f715166f700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f714c1602c0 0x7f714c071050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33522/0 (socket says 172.21.15.115:33522) 2023-03-25T00:57:38.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f7151e70700 1 -- 172.21.15.115:0/2873091526 learned_addr learned my addr 172.21.15.115:0/2873091526 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:38.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f7151e70700 1 -- 172.21.15.115:0/2873091526 >> v1:172.21.15.121:6789/0 conn(0x7f714c071760 legacy=0x7f714c0cdac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f7151e70700 1 -- 172.21.15.115:0/2873091526 >> v1:172.21.15.115:6789/0 conn(0x7f714c1602c0 legacy=0x7f714c071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.920 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f7151e70700 1 -- 172.21.15.115:0/2873091526 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f714c0b3ed0 con 0x7f714c0ce1d0 2023-03-25T00:57:38.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.914+0000 7f7151e70700 1 --2- 172.21.15.115:0/2873091526 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c0ce1d0 0x7f714c167870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7144009e50 tx=0x7f714400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dfd8d520a9d88c5 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f7143fff700 1 -- 172.21.15.115:0/2873091526 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714400e070 con 0x7f714c0ce1d0 2023-03-25T00:57:38.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f7143fff700 1 -- 172.21.15.115:0/2873091526 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714400b9c0 con 0x7f714c0ce1d0 2023-03-25T00:57:38.921 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f7143fff700 1 -- 172.21.15.115:0/2873091526 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f714400bd70 con 0x7f714c0ce1d0 2023-03-25T00:57:38.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f71538d3700 1 -- 172.21.15.115:0/2873091526 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c0ce1d0 msgr2=0x7f714c167870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f71538d3700 1 --2- 172.21.15.115:0/2873091526 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c0ce1d0 0x7f714c167870 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f7144009e50 tx=0x7f714400b330 comp rx=0 tx=0).stop 2023-03-25T00:57:38.922 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f71538d3700 1 -- 172.21.15.115:0/2873091526 shutdown_connections 2023-03-25T00:57:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f71538d3700 1 --2- 172.21.15.115:0/2873091526 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c0ce1d0 0x7f714c167870 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f71538d3700 1 -- 172.21.15.115:0/2873091526 >> 172.21.15.115:0/2873091526 conn(0x7f714c0cfef0 msgr2=0x7f714c0d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f71538d3700 1 -- 172.21.15.115:0/2873091526 shutdown_connections 2023-03-25T00:57:38.923 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.915+0000 7f71538d3700 1 -- 172.21.15.115:0/2873091526 wait complete. 2023-03-25T00:57:38.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.916+0000 7f71538d3700 1 Processor -- start 2023-03-25T00:57:38.924 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.916+0000 7f71538d3700 1 -- start start 2023-03-25T00:57:38.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f71538d3700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c071760 0x7f714c155920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f71538d3700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714c0ce1d0 0x7f714c158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.928 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f71538d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1602c0 0x7f714c157870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f71538d3700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c1567d0 con 0x7f714c071760 2023-03-25T00:57:38.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f71538d3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f714c156ab0 con 0x7f714c1602c0 2023-03-25T00:57:38.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f71538d3700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f714c156d90 con 0x7f714c0ce1d0 2023-03-25T00:57:38.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f715166f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c071760 0x7f714c155920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f715166f700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c071760 0x7f714c155920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32966/0 (socket says 172.21.15.115:32966) 2023-03-25T00:57:38.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f715166f700 1 -- 172.21.15.115:0/3624606748 learned_addr learned my addr 172.21.15.115:0/3624606748 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:38.929 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f7151e70700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1602c0 0x7f714c157870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f7150e6e700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714c0ce1d0 0x7f714c158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f715166f700 1 -- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714c0ce1d0 msgr2=0x7f714c158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.930 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f715166f700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714c0ce1d0 0x7f714c158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f715166f700 1 -- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1602c0 msgr2=0x7f714c157870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:38.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f715166f700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1602c0 0x7f714c157870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:38.931 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f715166f700 1 -- 172.21.15.115:0/3624606748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7144009b30 con 0x7f714c071760 2023-03-25T00:57:38.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f7150e6e700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714c0ce1d0 0x7f714c158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:38.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.917+0000 7f7151e70700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1602c0 0x7f714c157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-25T00:57:38.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.918+0000 7f715166f700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c071760 0x7f714c155920 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f713c00c500 tx=0x7f713c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.932 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.918+0000 7f7141ffb700 1 -- 172.21.15.115:0/3624606748 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c005cb0 con 0x7f714c071760 2023-03-25T00:57:38.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.918+0000 7f7141ffb700 1 -- 172.21.15.115:0/3624606748 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f713c005e10 con 0x7f714c071760 2023-03-25T00:57:38.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.918+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f714c0ce980 con 0x7f714c071760 2023-03-25T00:57:38.933 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.918+0000 7f7141ffb700 1 -- 172.21.15.115:0/3624606748 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f713c0072d0 con 0x7f714c071760 2023-03-25T00:57:38.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.918+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f714c0b41b0 con 0x7f714c071760 2023-03-25T00:57:38.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.919+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f714c0b59c0 con 0x7f714c071760 2023-03-25T00:57:38.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.919+0000 7f7141ffb700 1 -- 172.21.15.115:0/3624606748 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f713c007430 con 0x7f714c071760 2023-03-25T00:57:38.934 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.919+0000 7f7141ffb700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7130069de0 0x7f713006c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.920+0000 7f7141ffb700 1 -- 172.21.15.115:0/3624606748 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f713c086f00 con 0x7f714c071760 2023-03-25T00:57:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.920+0000 7f7150e6e700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7130069de0 0x7f713006c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.920+0000 7f7150e6e700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7130069de0 0x7f713006c290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f714c0cc1b0 tx=0x7f7148009040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:38.935 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:38.924+0000 7f7141ffb700 1 -- 172.21.15.115:0/3624606748 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f713c04c160 con 0x7f714c071760 2023-03-25T00:57:39.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.113+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f714c074580 con 0x7f714c071760 2023-03-25T00:57:39.116 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.114+0000 7f7141ffb700 1 -- 172.21.15.115:0/3624606748 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+126 (secure 0 0 0) 0x7f713c017090 con 0x7f714c071760 2023-03-25T00:57:39.116 INFO:teuthology.orchestra.run.smithi115.stdout: 2023-03-25T00:57:39.116 INFO:teuthology.orchestra.run.smithi115.stdout:[ 2023-03-25T00:57:39.117 INFO:teuthology.orchestra.run.smithi115.stdout: { 2023-03-25T00:57:39.117 INFO:teuthology.orchestra.run.smithi115.stdout: "poolnum": 1, 2023-03-25T00:57:39.117 INFO:teuthology.orchestra.run.smithi115.stdout: "poolname": "rbd" 2023-03-25T00:57:39.117 INFO:teuthology.orchestra.run.smithi115.stdout: }, 2023-03-25T00:57:39.117 INFO:teuthology.orchestra.run.smithi115.stdout: { 2023-03-25T00:57:39.117 INFO:teuthology.orchestra.run.smithi115.stdout: "poolnum": 2, 2023-03-25T00:57:39.117 INFO:teuthology.orchestra.run.smithi115.stdout: "poolname": ".mgr" 2023-03-25T00:57:39.118 INFO:teuthology.orchestra.run.smithi115.stdout: } 2023-03-25T00:57:39.118 INFO:teuthology.orchestra.run.smithi115.stdout:] 2023-03-25T00:57:39.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7130069de0 msgr2=0x7f713006c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:39.119 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7130069de0 0x7f713006c290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f714c0cc1b0 tx=0x7f7148009040 comp rx=0 tx=0).stop 2023-03-25T00:57:39.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c071760 msgr2=0x7f714c155920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:39.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c071760 0x7f714c155920 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f713c00c500 tx=0x7f713c00c8c0 comp rx=0 tx=0).stop 2023-03-25T00:57:39.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 shutdown_connections 2023-03-25T00:57:39.120 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f7130069de0 0x7f713006c290 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:39.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f714c1602c0 0x7f714c157870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:39.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f714c071760 0x7f714c155920 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:39.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 --2- 172.21.15.115:0/3624606748 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f714c0ce1d0 0x7f714c158b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:39.121 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 >> 172.21.15.115:0/3624606748 conn(0x7f714c0cfef0 msgr2=0x7f714c0b0290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:39.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 shutdown_connections 2023-03-25T00:57:39.122 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.116+0000 7f71538d3700 1 -- 172.21.15.115:0/3624606748 wait complete. 2023-03-25T00:57:39.130 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-25T00:57:39.130 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-25T00:57:39.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.254+0000 7f9765dba700 1 Processor -- start 2023-03-25T00:57:39.256 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.254+0000 7f9765dba700 1 -- start start 2023-03-25T00:57:39.258 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f9765dba700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f976015a620 0x7f97600c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:39.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f9765dba700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97600c7440 con 0x7f97600b4050 2023-03-25T00:57:39.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f9765dba700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97600ca250 con 0x7f976015a9f0 2023-03-25T00:57:39.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f9765dba700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f976015ef10 con 0x7f976015a620 2023-03-25T00:57:39.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f975ffff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f97600b4050 0x7f97600caf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33538/0 (socket says 172.21.15.115:33538) 2023-03-25T00:57:39.259 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f975ffff700 1 -- 172.21.15.115:0/3960123337 learned_addr learned my addr 172.21.15.115:0/3960123337 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:39.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f975f7fe700 1 --2- 172.21.15.115:0/3960123337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f976015a620 0x7f97600c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:39.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f975e7fc700 1 -- 172.21.15.115:0/3960123337 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269888289 0 0) 0x7f97600c7440 con 0x7f97600b4050 2023-03-25T00:57:39.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f975e7fc700 1 -- 172.21.15.115:0/3960123337 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97480035b0 con 0x7f97600b4050 2023-03-25T00:57:39.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f975e7fc700 1 -- 172.21.15.115:0/3960123337 <== mon.1 v1:172.21.15.121:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 950263922 0 0) 0x7f97600ca250 con 0x7f976015a9f0 2023-03-25T00:57:39.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.255+0000 7f975e7fc700 1 -- 172.21.15.115:0/3960123337 --> v1:172.21.15.121:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97600c7440 con 0x7f976015a9f0 2023-03-25T00:57:39.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f975f7fe700 1 -- 172.21.15.115:0/3960123337 >> v1:172.21.15.121:6789/0 conn(0x7f976015a9f0 legacy=0x7f976015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:39.260 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f975f7fe700 1 -- 172.21.15.115:0/3960123337 >> v1:172.21.15.115:6789/0 conn(0x7f97600b4050 legacy=0x7f97600caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f975e7fc700 1 -- 172.21.15.115:0/3960123337 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1643783204 0 0) 0x7f97480035b0 con 0x7f97600b4050 2023-03-25T00:57:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f975f7fe700 1 -- 172.21.15.115:0/3960123337 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f976015f1f0 con 0x7f976015a620 2023-03-25T00:57:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f975f7fe700 1 --2- 172.21.15.115:0/3960123337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f976015a620 0x7f97600c6f00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9754009870 tx=0x7f9754009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=795312a0948d4896 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f975e7fc700 1 -- 172.21.15.115:0/3960123337 <== mon.2 v2:172.21.15.121:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97580019f0 con 0x7f976015a620 2023-03-25T00:57:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f975e7fc700 1 -- 172.21.15.115:0/3960123337 <== mon.2 v2:172.21.15.121:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9754004030 con 0x7f976015a620 2023-03-25T00:57:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f975e7fc700 1 -- 172.21.15.115:0/3960123337 <== mon.2 v2:172.21.15.121:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97540043a0 con 0x7f976015a620 2023-03-25T00:57:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f9765dba700 1 -- 172.21.15.115:0/3960123337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f976015a620 msgr2=0x7f97600c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:39.261 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f9765dba700 1 --2- 172.21.15.115:0/3960123337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f976015a620 0x7f97600c6f00 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9754009870 tx=0x7f9754009b80 comp rx=0 tx=0).stop 2023-03-25T00:57:39.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f9765dba700 1 -- 172.21.15.115:0/3960123337 shutdown_connections 2023-03-25T00:57:39.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f9765dba700 1 --2- 172.21.15.115:0/3960123337 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f976015a620 0x7f97600c6f00 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:39.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.256+0000 7f9765dba700 1 -- 172.21.15.115:0/3960123337 >> 172.21.15.115:0/3960123337 conn(0x7f97600d0b50 msgr2=0x7f97600cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:39.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.257+0000 7f9765dba700 1 -- 172.21.15.115:0/3960123337 shutdown_connections 2023-03-25T00:57:39.262 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.257+0000 7f9765dba700 1 -- 172.21.15.115:0/3960123337 wait complete. 2023-03-25T00:57:39.264 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.258+0000 7f9765dba700 1 Processor -- start 2023-03-25T00:57:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.258+0000 7f9765dba700 1 -- start start 2023-03-25T00:57:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9765dba700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97600b4050 0x7f9760145c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9765dba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f976015a620 0x7f9760146ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9765dba700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f976015a9f0 0x7f97600b1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9765dba700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760146180 con 0x7f976015a9f0 2023-03-25T00:57:39.265 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9765dba700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9760147810 con 0x7f976015a620 2023-03-25T00:57:39.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9765dba700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9760147af0 con 0x7f97600b4050 2023-03-25T00:57:39.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f976015a9f0 0x7f97600b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:39.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f976015a9f0 0x7f97600b1090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32984/0 (socket says 172.21.15.115:32984) 2023-03-25T00:57:39.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 -- 172.21.15.115:0/1514123299 learned_addr learned my addr 172.21.15.115:0/1514123299 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:39.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975f7fe700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97600b4050 0x7f9760145c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:39.266 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975effd700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f976015a620 0x7f9760146ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:39.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 -- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97600b4050 msgr2=0x7f9760145c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:39.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97600b4050 0x7f9760145c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:39.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 -- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f976015a620 msgr2=0x7f9760146ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:39.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f976015a620 0x7f9760146ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:39.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 -- 172.21.15.115:0/1514123299 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97540094a0 con 0x7f976015a9f0 2023-03-25T00:57:39.267 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975f7fe700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97600b4050 0x7f9760145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:39.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975effd700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f976015a620 0x7f9760146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:39.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f975ffff700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f976015a9f0 0x7f97600b1090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f975800cc70 tx=0x7f975800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:39.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9747fff700 1 -- 172.21.15.115:0/1514123299 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f975800ade0 con 0x7f976015a9f0 2023-03-25T00:57:39.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9747fff700 1 -- 172.21.15.115:0/1514123299 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f975800ce70 con 0x7f976015a9f0 2023-03-25T00:57:39.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97600ca760 con 0x7f976015a9f0 2023-03-25T00:57:39.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9747fff700 1 -- 172.21.15.115:0/1514123299 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97580072f0 con 0x7f976015a9f0 2023-03-25T00:57:39.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.259+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97600ca480 con 0x7f976015a9f0 2023-03-25T00:57:39.268 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.260+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97600caa40 con 0x7f976015a9f0 2023-03-25T00:57:39.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.261+0000 7f9747fff700 1 -- 172.21.15.115:0/1514123299 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f9758007450 con 0x7f976015a9f0 2023-03-25T00:57:39.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.261+0000 7f9747fff700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f974c069e10 0x7f974c06c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:39.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.261+0000 7f9747fff700 1 -- 172.21.15.115:0/1514123299 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f9758087c20 con 0x7f976015a9f0 2023-03-25T00:57:39.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.261+0000 7f975f7fe700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f974c069e10 0x7f974c06c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:39.269 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.261+0000 7f975f7fe700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f974c069e10 0x7f974c06c2c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f97540090b0 tx=0x7f9754008030 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:39.270 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.265+0000 7f9747fff700 1 -- 172.21.15.115:0/1514123299 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f9758053820 con 0x7f976015a9f0 2023-03-25T00:57:39.458 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:39.456+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f97600cd140 con 0x7f976015a9f0 2023-03-25T00:57:40.405 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.403+0000 7f9747fff700 1 -- 172.21.15.115:0/1514123299 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f97580180b0 con 0x7f976015a9f0 2023-03-25T00:57:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f974c069e10 msgr2=0x7f974c06c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f974c069e10 0x7f974c06c2c0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f97540090b0 tx=0x7f9754008030 comp rx=0 tx=0).stop 2023-03-25T00:57:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f976015a9f0 msgr2=0x7f97600b1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:40.410 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f976015a9f0 0x7f97600b1090 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f975800cc70 tx=0x7f975800a340 comp rx=0 tx=0).stop 2023-03-25T00:57:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 shutdown_connections 2023-03-25T00:57:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f974c069e10 0x7f974c06c2c0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f976015a9f0 0x7f97600b1090 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f97600b4050 0x7f9760145c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 --2- 172.21.15.115:0/1514123299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f976015a620 0x7f9760146ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 >> 172.21.15.115:0/1514123299 conn(0x7f97600d0b50 msgr2=0x7f97600c94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:40.411 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 shutdown_connections 2023-03-25T00:57:40.412 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.408+0000 7f9765dba700 1 -- 172.21.15.115:0/1514123299 wait complete. 2023-03-25T00:57:40.412 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_metadata' created 2023-03-25T00:57:40.426 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-03-25T00:57:40.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.559+0000 7f4e7a672700 1 Processor -- start 2023-03-25T00:57:40.561 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.559+0000 7f4e7a672700 1 -- start start 2023-03-25T00:57:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.559+0000 7f4e7a672700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4e740ced00 0x7f4e74070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.559+0000 7f4e7a672700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7415d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.559+0000 7f4e7a672700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e74160610 con 0x7f4e740cf7e0 2023-03-25T00:57:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.559+0000 7f4e7a672700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e740cf170 con 0x7f4e740ced00 2023-03-25T00:57:40.563 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e7a672700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e740719a0 con 0x7f4e74071520 2023-03-25T00:57:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e737fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4e740ced00 0x7f4e74070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e737fe700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f4e740ced00 0x7f4e74070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32990/0 (socket says 172.21.15.115:32990) 2023-03-25T00:57:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e73fff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f4e740cf7e0 0x7f4e740cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33548/0 (socket says 172.21.15.115:33548) 2023-03-25T00:57:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e737fe700 1 -- 172.21.15.115:0/727055974 learned_addr learned my addr 172.21.15.115:0/727055974 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e78c0f700 1 --2- 172.21.15.115:0/727055974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7415d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e737fe700 1 -- 172.21.15.115:0/727055974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 msgr2=0x7f4e7415d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:40.564 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e737fe700 1 --2- 172.21.15.115:0/727055974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7415d370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e737fe700 1 -- 172.21.15.115:0/727055974 >> v1:172.21.15.115:6789/0 conn(0x7f4e740cf7e0 legacy=0x7f4e740cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:40.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e737fe700 1 -- 172.21.15.115:0/727055974 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e741608f0 con 0x7f4e740ced00 2023-03-25T00:57:40.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e72ffd700 1 -- 172.21.15.115:0/727055974 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96777792 0 0) 0x7f4e74160610 con 0x7f4e740cf7e0 2023-03-25T00:57:40.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e78c0f700 1 --2- 172.21.15.115:0/727055974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7415d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:40.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e737fe700 1 --2- 172.21.15.115:0/727055974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e74070fe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e640098f0 tx=0x7f4e64009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b58fb9c2079cbf81 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:40.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e72ffd700 1 -- 172.21.15.115:0/727055974 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6400b8a0 con 0x7f4e740ced00 2023-03-25T00:57:40.565 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e72ffd700 1 -- 172.21.15.115:0/727055974 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6400ba00 con 0x7f4e740ced00 2023-03-25T00:57:40.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.560+0000 7f4e72ffd700 1 -- 172.21.15.115:0/727055974 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e6400bd90 con 0x7f4e740ced00 2023-03-25T00:57:40.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.561+0000 7f4e7a672700 1 -- 172.21.15.115:0/727055974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 msgr2=0x7f4e74070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:40.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.561+0000 7f4e7a672700 1 --2- 172.21.15.115:0/727055974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e74070fe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f4e640098f0 tx=0x7f4e64009c00 comp rx=0 tx=0).stop 2023-03-25T00:57:40.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.561+0000 7f4e7a672700 1 -- 172.21.15.115:0/727055974 shutdown_connections 2023-03-25T00:57:40.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.561+0000 7f4e7a672700 1 --2- 172.21.15.115:0/727055974 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7415d370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.561+0000 7f4e7a672700 1 --2- 172.21.15.115:0/727055974 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e74070fe0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.561+0000 7f4e7a672700 1 -- 172.21.15.115:0/727055974 >> 172.21.15.115:0/727055974 conn(0x7f4e740d0b50 msgr2=0x7f4e740cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:40.566 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.561+0000 7f4e7a672700 1 -- 172.21.15.115:0/727055974 shutdown_connections 2023-03-25T00:57:40.567 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.561+0000 7f4e7a672700 1 -- 172.21.15.115:0/727055974 wait complete. 2023-03-25T00:57:40.568 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.562+0000 7f4e7a672700 1 Processor -- start 2023-03-25T00:57:40.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.562+0000 7f4e7a672700 1 -- start start 2023-03-25T00:57:40.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e7a672700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7414f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:40.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e7a672700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e741502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:40.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e7a672700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e740cf7e0 0x7f4e74150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:40.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e7a672700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e74150d60 con 0x7f4e740ced00 2023-03-25T00:57:40.569 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e7a672700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e741f2370 con 0x7f4e740cf7e0 2023-03-25T00:57:40.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e7a672700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7414f890 con 0x7f4e74071520 2023-03-25T00:57:40.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e741502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:40.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e741502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:32994/0 (socket says 172.21.15.115:32994) 2023-03-25T00:57:40.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 -- 172.21.15.115:0/1953392350 learned_addr learned my addr 172.21.15.115:0/1953392350 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:40.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e73fff700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7414f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:40.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e78c0f700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e740cf7e0 0x7f4e74150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:40.570 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 -- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 msgr2=0x7f4e7414f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:40.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7414f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 -- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e740cf7e0 msgr2=0x7f4e74150820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:40.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e740cf7e0 0x7f4e74150820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:40.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 -- 172.21.15.115:0/1953392350 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e64009550 con 0x7f4e740ced00 2023-03-25T00:57:40.571 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e78c0f700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e740cf7e0 0x7f4e74150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-25T00:57:40.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.563+0000 7f4e737fe700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e741502e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009c40 tx=0x7f4e64004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:40.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.564+0000 7f4e70ff9700 1 -- 172.21.15.115:0/1953392350 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e640036a0 con 0x7f4e740ced00 2023-03-25T00:57:40.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.564+0000 7f4e70ff9700 1 -- 172.21.15.115:0/1953392350 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e64003800 con 0x7f4e740ced00 2023-03-25T00:57:40.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.564+0000 7f4e70ff9700 1 -- 172.21.15.115:0/1953392350 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e64003b70 con 0x7f4e740ced00 2023-03-25T00:57:40.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.564+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e74160eb0 con 0x7f4e740ced00 2023-03-25T00:57:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.564+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e74160bd0 con 0x7f4e740ced00 2023-03-25T00:57:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.564+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e74161190 con 0x7f4e740ced00 2023-03-25T00:57:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.565+0000 7f4e70ff9700 1 -- 172.21.15.115:0/1953392350 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f4e64003cd0 con 0x7f4e740ced00 2023-03-25T00:57:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.565+0000 7f4e70ff9700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f4e60069de0 0x7f4e6006c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.565+0000 7f4e70ff9700 1 -- 172.21.15.115:0/1953392350 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5580+0+0 (secure 0 0 0) 0x7f4e64021030 con 0x7f4e740ced00 2023-03-25T00:57:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.565+0000 7f4e73fff700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f4e60069de0 0x7f4e6006c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:40.573 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.566+0000 7f4e73fff700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f4e60069de0 0x7f4e6006c290 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e68000f60 tx=0x7f4e68007e70 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:40.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.570+0000 7f4e70ff9700 1 -- 172.21.15.115:0/1953392350 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f4e64055130 con 0x7f4e740ced00 2023-03-25T00:57:40.759 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:40.757+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f4e740ceae0 con 0x7f4e740ced00 2023-03-25T00:57:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.413+0000 7f4e70ff9700 1 -- 172.21.15.115:0/1953392350 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v18) v1 ==== 171+0+0 (secure 0 0 0) 0x7f4e64058750 con 0x7f4e740ced00 2023-03-25T00:57:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f4e60069de0 msgr2=0x7f4e6006c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.420 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f4e60069de0 0x7f4e6006c290 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e68000f60 tx=0x7f4e68007e70 comp rx=0 tx=0).stop 2023-03-25T00:57:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 msgr2=0x7f4e741502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e741502e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009c40 tx=0x7f4e64004750 comp rx=0 tx=0).stop 2023-03-25T00:57:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 shutdown_connections 2023-03-25T00:57:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f4e60069de0 0x7f4e6006c290 secure :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e68000f60 tx=0x7f4e68007e70 comp rx=0 tx=0).stop 2023-03-25T00:57:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4e740cf7e0 0x7f4e74150820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f4e74071520 0x7f4e7414f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.421 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 --2- 172.21.15.115:0/1953392350 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f4e740ced00 0x7f4e741502e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.422 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 >> 172.21.15.115:0/1953392350 conn(0x7f4e740d0b50 msgr2=0x7f4e740b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:41.426 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.418+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 shutdown_connections 2023-03-25T00:57:41.427 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.419+0000 7f4e7a672700 1 -- 172.21.15.115:0/1953392350 wait complete. 2023-03-25T00:57:41.427 INFO:teuthology.orchestra.run.smithi115.stderr:pool 'cephfs_data' created 2023-03-25T00:57:41.443 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-25T00:57:41.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.570+0000 7f532e183700 1 Processor -- start 2023-03-25T00:57:41.572 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.570+0000 7f532e183700 1 -- start start 2023-03-25T00:57:41.574 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.570+0000 7f532e183700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 0x7f53280c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.570+0000 7f532e183700 1 -- --> v1:172.21.15.115:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53280c7440 con 0x7f53280b4050 2023-03-25T00:57:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.570+0000 7f532e183700 1 -- --> v1:172.21.15.121:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53280ca250 con 0x7f532815a620 2023-03-25T00:57:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.570+0000 7f532e183700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f532815ef10 con 0x7f532815a9f0 2023-03-25T00:57:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f5327fff700 1 --1- >> v1:172.21.15.115:6789/0 conn(0x7f53280b4050 0x7f53280caf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.115:6789/0 says I am v1:172.21.15.115:33566/0 (socket says 172.21.15.115:33566) 2023-03-25T00:57:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f5327fff700 1 -- 172.21.15.115:0/2556529318 learned_addr learned my addr 172.21.15.115:0/2556529318 (peer_addr_for_me v1:172.21.15.115:0/0) 2023-03-25T00:57:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f5326ffd700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 0x7f53280c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.575 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 <== mon.0 v1:172.21.15.115:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382203321 0 0) 0x7f53280c7440 con 0x7f53280b4050 2023-03-25T00:57:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 --> v1:172.21.15.115:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53140035b0 con 0x7f53280b4050 2023-03-25T00:57:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 <== mon.0 v1:172.21.15.115:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2158353954 0 0) 0x7f53140035b0 con 0x7f53280b4050 2023-03-25T00:57:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 msgr2=0x7f53280c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 0x7f53280c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 >> v1:172.21.15.121:6789/0 conn(0x7f532815a620 legacy=0x7f532815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 --> v1:172.21.15.115:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f532815f1f0 con 0x7f53280b4050 2023-03-25T00:57:41.576 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 <== mon.0 v1:172.21.15.115:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 2816979759 0 0) 0x7f53200019f0 con 0x7f53280b4050 2023-03-25T00:57:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 >> v1:172.21.15.115:6789/0 conn(0x7f53280b4050 legacy=0x7f53280caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 0x7f5314003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5314003f70 0x7f5314004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5314004d90 0x7f5314005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532815ef10 con 0x7f5314004d90 2023-03-25T00:57:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531400ba40 con 0x7f5314003f70 2023-03-25T00:57:41.577 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f531400bc00 con 0x7f532815a9f0 2023-03-25T00:57:41.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f5326ffd700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5314003f70 0x7f5314004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f53277fe700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 0x7f5314003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.571+0000 7f5327fff700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5314004d90 0x7f5314005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f5327fff700 1 -- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 msgr2=0x7f5314003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:41.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f5327fff700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 0x7f5314003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f5327fff700 1 -- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5314003f70 msgr2=0x7f5314004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:41.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f5327fff700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5314003f70 0x7f5314004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.578 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f5327fff700 1 -- 172.21.15.115:0/2556529318 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53140035b0 con 0x7f5314004d90 2023-03-25T00:57:41.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f53277fe700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f532815a9f0 0x7f5314003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-25T00:57:41.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f5327fff700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5314004d90 0x7f5314005200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f532000a830 tx=0x7f5314005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6059cc1e4d6a05bf server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:41.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320012600 con 0x7f5314004d90 2023-03-25T00:57:41.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53200127a0 con 0x7f5314004d90 2023-03-25T00:57:41.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f53267fc700 1 -- 172.21.15.115:0/2556529318 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5320012b50 con 0x7f5314004d90 2023-03-25T00:57:41.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f532e183700 1 -- 172.21.15.115:0/2556529318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5314004d90 msgr2=0x7f5314005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f532e183700 1 --2- 172.21.15.115:0/2556529318 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f5314004d90 0x7f5314005200 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f532000a830 tx=0x7f5314005880 comp rx=0 tx=0).stop 2023-03-25T00:57:41.579 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f5327fff700 1 -- 172.21.15.115:0/2556529318 reap_dead start 2023-03-25T00:57:41.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f532e183700 1 -- 172.21.15.115:0/2556529318 shutdown_connections 2023-03-25T00:57:41.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f532e183700 1 -- 172.21.15.115:0/2556529318 >> 172.21.15.115:0/2556529318 conn(0x7f53280d0b50 msgr2=0x7f53280cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:41.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f532e183700 1 -- 172.21.15.115:0/2556529318 shutdown_connections 2023-03-25T00:57:41.580 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.572+0000 7f532e183700 1 -- 172.21.15.115:0/2556529318 wait complete. 2023-03-25T00:57:41.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.574+0000 7f532e183700 1 Processor -- start 2023-03-25T00:57:41.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.574+0000 7f532e183700 1 -- start start 2023-03-25T00:57:41.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.574+0000 7f532e183700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53281efb80 0x7f53280af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.582 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.574+0000 7f532e183700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f532806cf30 0x7f532806d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.574+0000 7f532e183700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53281f3810 0x7f53281f3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.574+0000 7f532e183700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f532806d8e0 con 0x7f53281efb80 2023-03-25T00:57:41.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.574+0000 7f532e183700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53280b0f20 con 0x7f532806cf30 2023-03-25T00:57:41.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.574+0000 7f532e183700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53280b0310 con 0x7f53281f3810 2023-03-25T00:57:41.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53281efb80 0x7f53280af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53281efb80 0x7f53280af9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33014/0 (socket says 172.21.15.115:33014) 2023-03-25T00:57:41.583 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 -- 172.21.15.115:0/1110738499 learned_addr learned my addr 172.21.15.115:0/1110738499 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:41.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f5327fff700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53281f3810 0x7f53281f3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 -- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53281f3810 msgr2=0x7f53281f3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53281f3810 0x7f53281f3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 -- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f532806cf30 msgr2=0x7f532806d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f5326ffd700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f532806cf30 0x7f532806d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f532806cf30 0x7f532806d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 -- 172.21.15.115:0/1110738499 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53280ca760 con 0x7f53281efb80 2023-03-25T00:57:41.584 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f5327fff700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53281f3810 0x7f53281f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-25T00:57:41.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f53277fe700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53281efb80 0x7f53280af9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f531000fb30 tx=0x7f531000fef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:41.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f531ffff700 1 -- 172.21.15.115:0/1110738499 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53100052e0 con 0x7f53281efb80 2023-03-25T00:57:41.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f531ffff700 1 -- 172.21.15.115:0/1110738499 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5310005440 con 0x7f53281efb80 2023-03-25T00:57:41.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53280ca480 con 0x7f53281efb80 2023-03-25T00:57:41.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f531ffff700 1 -- 172.21.15.115:0/1110738499 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53100038c0 con 0x7f53281efb80 2023-03-25T00:57:41.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.575+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53280caa40 con 0x7f53281efb80 2023-03-25T00:57:41.585 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.576+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5328159de0 con 0x7f53281efb80 2023-03-25T00:57:41.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.577+0000 7f531ffff700 1 -- 172.21.15.115:0/1110738499 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f5310003a20 con 0x7f53281efb80 2023-03-25T00:57:41.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.577+0000 7f531ffff700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5308069de0 0x7f530806c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.577+0000 7f5326ffd700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5308069de0 0x7f530806c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.577+0000 7f531ffff700 1 -- 172.21.15.115:0/1110738499 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f5310088070 con 0x7f53281efb80 2023-03-25T00:57:41.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.577+0000 7f5326ffd700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5308069de0 0x7f530806c290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f532815ee40 tx=0x7f5318008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:41.586 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.582+0000 7f531ffff700 1 -- 172.21.15.115:0/1110738499 <== mon.0 v2:172.21.15.115:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f5310053980 con 0x7f53281efb80 2023-03-25T00:57:41.814 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.812+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f53281560c0 con 0x7f53281efb80 2023-03-25T00:57:41.821 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.819+0000 7f531ffff700 1 -- 172.21.15.115:0/1110738499 <== mon.0 v2:172.21.15.115:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7f5310056fa0 con 0x7f53281efb80 2023-03-25T00:57:41.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.826+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5308069de0 msgr2=0x7f530806c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.826+0000 7f532e183700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5308069de0 0x7f530806c290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f532815ee40 tx=0x7f5318008040 comp rx=0 tx=0).stop 2023-03-25T00:57:41.829 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.826+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53281efb80 msgr2=0x7f53280af9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.830 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.826+0000 7f532e183700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53281efb80 0x7f53280af9a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f531000fb30 tx=0x7f531000fef0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.829+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 shutdown_connections 2023-03-25T00:57:41.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.829+0000 7f532e183700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f5308069de0 0x7f530806c290 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.829+0000 7f532e183700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f53281f3810 0x7f53281f3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.829+0000 7f532e183700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f53281efb80 0x7f53280af9a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.831 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.829+0000 7f532e183700 1 --2- 172.21.15.115:0/1110738499 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f532806cf30 0x7f532806d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.829+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 >> 172.21.15.115:0/1110738499 conn(0x7f53280d0b50 msgr2=0x7f53280c7710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:41.832 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.830+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 shutdown_connections 2023-03-25T00:57:41.833 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.831+0000 7f532e183700 1 -- 172.21.15.115:0/1110738499 wait complete. 2023-03-25T00:57:41.836 INFO:teuthology.orchestra.run.smithi115.stderr:new fs with metadata pool 3 and data pool 4 2023-03-25T00:57:41.852 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2023-03-25T00:57:41.853 DEBUG:teuthology.orchestra.run.smithi115:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2023-03-25T00:57:41.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.975+0000 7f14394f1700 1 Processor -- start 2023-03-25T00:57:41.977 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.975+0000 7f14394f1700 1 -- start start 2023-03-25T00:57:41.979 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14394f1700 1 --2- >> v2:172.21.15.121:3300/0 conn(0x7f1434156ef0 0x7f143406ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14394f1700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f14340b5500 0x7f143406ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14394f1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 0x7f1434160590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14394f1700 1 -- --> v2:172.21.15.115:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f143415df70 con 0x7f14340b5500 2023-03-25T00:57:41.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14394f1700 1 -- --> v2:172.21.15.121:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f143406b790 con 0x7f1434156ef0 2023-03-25T00:57:41.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14394f1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14340d29f0 con 0x7f143406b350 2023-03-25T00:57:41.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f14340b5500 0x7f143406ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.980 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 --2- >> v2:172.21.15.115:3300/0 conn(0x7f14340b5500 0x7f143406ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33024/0 (socket says 172.21.15.115:33024) 2023-03-25T00:57:41.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 -- 172.21.15.115:0/4075368648 learned_addr learned my addr 172.21.15.115:0/4075368648 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:41.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14337fe700 1 --2- 172.21.15.115:0/4075368648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 0x7f1434160590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f1432ffd700 1 --2- 172.21.15.115:0/4075368648 >> v2:172.21.15.121:3300/0 conn(0x7f1434156ef0 0x7f143406ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 -- 172.21.15.115:0/4075368648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 msgr2=0x7f1434160590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:41.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 --2- 172.21.15.115:0/4075368648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 0x7f1434160590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 -- 172.21.15.115:0/4075368648 >> v2:172.21.15.121:3300/0 conn(0x7f1434156ef0 msgr2=0x7f143406ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-25T00:57:41.981 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 --2- 172.21.15.115:0/4075368648 >> v2:172.21.15.121:3300/0 conn(0x7f1434156ef0 0x7f143406ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 -- 172.21.15.115:0/4075368648 --> v2:172.21.15.115:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143415d2f0 con 0x7f14340b5500 2023-03-25T00:57:41.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f14327fc700 1 --2- 172.21.15.115:0/4075368648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14340b5500 0x7f143406ae10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f143408be90 tx=0x7f1428009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6bec3501119c17a server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:41.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f1431ffb700 1 -- 172.21.15.115:0/4075368648 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142800b870 con 0x7f14340b5500 2023-03-25T00:57:41.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f1431ffb700 1 -- 172.21.15.115:0/4075368648 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142800b9d0 con 0x7f14340b5500 2023-03-25T00:57:41.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.976+0000 7f1431ffb700 1 -- 172.21.15.115:0/4075368648 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142800bd60 con 0x7f14340b5500 2023-03-25T00:57:41.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 -- 172.21.15.115:0/4075368648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14340b5500 msgr2=0x7f143406ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 --2- 172.21.15.115:0/4075368648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14340b5500 0x7f143406ae10 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f143408be90 tx=0x7f1428009bd0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.982 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 -- 172.21.15.115:0/4075368648 shutdown_connections 2023-03-25T00:57:41.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 --2- 172.21.15.115:0/4075368648 >> v2:172.21.15.121:3300/0 conn(0x7f1434156ef0 0x7f143406ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 --2- 172.21.15.115:0/4075368648 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 0x7f1434160590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 --2- 172.21.15.115:0/4075368648 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f14340b5500 0x7f143406ae10 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 -- 172.21.15.115:0/4075368648 >> 172.21.15.115:0/4075368648 conn(0x7f14340d3f90 msgr2=0x7f14340b4df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-25T00:57:41.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 -- 172.21.15.115:0/4075368648 shutdown_connections 2023-03-25T00:57:41.983 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.977+0000 7f14394f1700 1 -- 172.21.15.115:0/4075368648 wait complete. 2023-03-25T00:57:41.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.979+0000 7f14394f1700 1 Processor -- start 2023-03-25T00:57:41.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.979+0000 7f14394f1700 1 -- start start 2023-03-25T00:57:41.985 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14394f1700 1 --2- >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 0x7f14340c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14394f1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14340b5500 0x7f14341f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14394f1700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1434156ef0 0x7f14341f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14394f1700 1 -- --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14340cd200 con 0x7f1434156ef0 2023-03-25T00:57:41.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14394f1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14340ca1d0 con 0x7f14340b5500 2023-03-25T00:57:41.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14394f1700 1 -- --> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14340c9930 con 0x7f143406b350 2023-03-25T00:57:41.986 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1434156ef0 0x7f14341f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 --2- >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1434156ef0 0x7f14341f2ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.115:3300/0 says I am v2:172.21.15.115:33032/0 (socket says 172.21.15.115:33032) 2023-03-25T00:57:41.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 -- 172.21.15.115:0/1722163828 learned_addr learned my addr 172.21.15.115:0/1722163828 (peer_addr_for_me v2:172.21.15.115:0/0) 2023-03-25T00:57:41.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f1432ffd700 1 --2- 172.21.15.115:0/1722163828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 0x7f14340c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14327fc700 1 --2- 172.21.15.115:0/1722163828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14340b5500 0x7f14341f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 -- 172.21.15.115:0/1722163828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 msgr2=0x7f14340c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.987 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 --2- 172.21.15.115:0/1722163828 >> [v2:172.21.15.121:3301/0,v1:172.21.15.121:6790/0] conn(0x7f143406b350 0x7f14340c6e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 -- 172.21.15.115:0/1722163828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14340b5500 msgr2=0x7f14341f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-25T00:57:41.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 --2- 172.21.15.115:0/1722163828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f14340b5500 0x7f14341f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-25T00:57:41.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 -- 172.21.15.115:0/1722163828 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1428009550 con 0x7f1434156ef0 2023-03-25T00:57:41.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14337fe700 1 --2- 172.21.15.115:0/1722163828 >> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] conn(0x7f1434156ef0 0x7f14341f2ae0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f142c007e60 tx=0x7f142c00e3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:41.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f141b7fe700 1 -- 172.21.15.115:0/1722163828 <== mon.0 v2:172.21.15.115:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c00ed30 con 0x7f1434156ef0 2023-03-25T00:57:41.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f141b7fe700 1 -- 172.21.15.115:0/1722163828 <== mon.0 v2:172.21.15.115:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c014420 con 0x7f1434156ef0 2023-03-25T00:57:41.988 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14394f1700 1 -- 172.21.15.115:0/1722163828 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143415d8b0 con 0x7f1434156ef0 2023-03-25T00:57:41.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f141b7fe700 1 -- 172.21.15.115:0/1722163828 <== mon.0 v2:172.21.15.115:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f142c0147d0 con 0x7f1434156ef0 2023-03-25T00:57:41.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.980+0000 7f14394f1700 1 -- 172.21.15.115:0/1722163828 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143415d5d0 con 0x7f1434156ef0 2023-03-25T00:57:41.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.981+0000 7f14394f1700 1 -- 172.21.15.115:0/1722163828 --> [v2:172.21.15.115:3300/0,v1:172.21.15.115:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143415db90 con 0x7f1434156ef0 2023-03-25T00:57:41.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.982+0000 7f141b7fe700 1 -- 172.21.15.115:0/1722163828 <== mon.0 v2:172.21.15.115:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f142c014930 con 0x7f1434156ef0 2023-03-25T00:57:41.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.982+0000 7f141b7fe700 1 --2- 172.21.15.115:0/1722163828 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1420069de0 0x7f142006c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-25T00:57:41.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.982+0000 7f141b7fe700 1 -- 172.21.15.115:0/1722163828 <== mon.0 v2:172.21.15.115:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 6062+0+0 (secure 0 0 0) 0x7f142c0893c0 con 0x7f1434156ef0 2023-03-25T00:57:41.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.982+0000 7f1432ffd700 1 --2- 172.21.15.115:0/1722163828 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1420069de0 0x7f142006c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-25T00:57:41.989 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.983+0000 7f1432ffd700 1 --2- 172.21.15.115:0/1722163828 >> [v2:172.21.15.115:6832/97372,v1:172.21.15.115:6833/97372] conn(0x7f1420069de0 0x7f142006c290 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f1424006b20 tx=0x7f1424008040 comp rx=0 tx=0).ready entity=mgr.4104 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-25T00:57:41.990 INFO:teuthology.orchestra.run.smithi115.stderr:2023-03-25T00:57:41.986+0000 7f141b7fe700 1 -- 172.21.15.115:0/1722163828 <== mon.0 v2:172.21.15.115:3300/0