2023-03-24T23:09:12.863 INFO:root:teuthology version: 0.0.1.dev1+ge372916 2023-03-24T23:09:12.864 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi --verbose --archive /home/teuthworker/archive/pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219622 --owner scheduled_pdonnell@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} -- /home/teuthworker/archive/pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219622/orig.config.yaml 2023-03-24T23:09:12.908 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2023-03-24T23:09:12.993 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219622 branch: wip-pdonnell-testing-20230323.162417 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub} email: pdonnell@redhat.com first_in_suite: false job_id: '7219622' kernel: client: sha1: distro kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: rhel os_version: '8.6' overrides: admin_socket: branch: wip-pdonnell-testing-20230323.162417 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - inode wrongly marked free - bad backtrace on inode - inode table repaired for inode - Scrub error on inode - Scrub error on dir - Metadata damage detected sha1: c19e8f2a497e95f75abc43ec14bc6095e30dd2cb ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c19e8f2a497e95f75abc43ec14bc6095e30dd2cb kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20230323.162417 sha1: 568504667cd0ede116ce6b46562b9a2881100488 owner: scheduled_pdonnell@teuthology priority: 55 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 sha1: c19e8f2a497e95f75abc43ec14bc6095e30dd2cb sleep_before_teardown: 0 suite: fs suite_branch: wip-pdonnell-testing-20230323.162417 suite_path: /home/teuthworker/src/github.com_batrick_ceph_568504667cd0ede116ce6b46562b9a2881100488/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: 568504667cd0ede116ce6b46562b9a2881100488 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi191.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: null - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_forward_scrub teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: parallel-gzip teuthology_sha1: fe9b6e52013911547168fef4a8506325b6c76139 timestamp: 2023-03-24_22:00:37 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4538 2023-03-24T23:09:12.993 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_568504667cd0ede116ce6b46562b9a2881100488/qa; will attempt to use it 2023-03-24T23:09:12.994 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_568504667cd0ede116ce6b46562b9a2881100488/qa/tasks 2023-03-24T23:09:12.995 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-03-24T23:09:12.996 INFO:teuthology.task.internal:Checking packages... 2023-03-24T23:09:13.025 INFO:teuthology.task.internal:Checking packages for os_type 'rhel', flavor 'default' and ceph hash 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb' 2023-03-24T23:09:13.026 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-03-24T23:09:13.026 INFO:teuthology.packaging:ref: None 2023-03-24T23:09:13.026 INFO:teuthology.packaging:tag: None 2023-03-24T23:09:13.027 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20230323.162417 2023-03-24T23:09:13.027 INFO:teuthology.packaging:sha1: c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-24T23:09:13.027 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-pdonnell-testing-20230323.162417 2023-03-24T23:09:13.166 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-3124.gc19e8f2a 2023-03-24T23:09:13.166 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-03-24T23:09:13.173 INFO:teuthology.task.internal:no buildpackages task found 2023-03-24T23:09:13.174 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-03-24T23:09:13.207 INFO:teuthology.task.internal:Saving configuration 2023-03-24T23:09:13.223 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-03-24T23:09:13.257 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-03-24T23:09:13.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi191.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-24 23:04:36.196592', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-24T23:09:13.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'rhel', 'os_version': '8.6', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-03-24 23:04:36.198260', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='} 2023-03-24T23:09:13.313 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-03-24T23:09:13.320 INFO:teuthology.task.internal:roles: ubuntu@smithi191.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-03-24T23:09:13.320 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-03-24T23:09:13.320 INFO:teuthology.run_tasks:Running task console_log... 2023-03-24T23:09:13.472 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1e4434b5e0>, signals=[15]) 2023-03-24T23:09:13.474 INFO:teuthology.run_tasks:Running task internal.connect... 2023-03-24T23:09:13.481 INFO:teuthology.task.internal:Opening connections... 2023-03-24T23:09:13.481 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi191.front.sepia.ceph.com 2023-03-24T23:09:13.483 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-24T23:09:13.552 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2023-03-24T23:09:13.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-24T23:09:13.618 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-03-24T23:09:13.625 DEBUG:teuthology.orchestra.run.smithi191:> uname -m 2023-03-24T23:09:13.644 INFO:teuthology.orchestra.run.smithi191.stdout:x86_64 2023-03-24T23:09:13.644 DEBUG:teuthology.orchestra.run.smithi191:> cat /etc/os-release 2023-03-24T23:09:13.700 INFO:teuthology.orchestra.run.smithi191.stdout:NAME="Red Hat Enterprise Linux" 2023-03-24T23:09:13.700 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION="8.6 (Ootpa)" 2023-03-24T23:09:13.701 INFO:teuthology.orchestra.run.smithi191.stdout:ID="rhel" 2023-03-24T23:09:13.701 INFO:teuthology.orchestra.run.smithi191.stdout:ID_LIKE="fedora" 2023-03-24T23:09:13.701 INFO:teuthology.orchestra.run.smithi191.stdout:VERSION_ID="8.6" 2023-03-24T23:09:13.701 INFO:teuthology.orchestra.run.smithi191.stdout:PLATFORM_ID="platform:el8" 2023-03-24T23:09:13.701 INFO:teuthology.orchestra.run.smithi191.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-24T23:09:13.701 INFO:teuthology.orchestra.run.smithi191.stdout:ANSI_COLOR="0;31" 2023-03-24T23:09:13.701 INFO:teuthology.orchestra.run.smithi191.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-24T23:09:13.702 INFO:teuthology.orchestra.run.smithi191.stdout:HOME_URL="https://www.redhat.com/" 2023-03-24T23:09:13.702 INFO:teuthology.orchestra.run.smithi191.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-24T23:09:13.702 INFO:teuthology.orchestra.run.smithi191.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-24T23:09:13.702 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:09:13.702 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-24T23:09:13.702 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-24T23:09:13.702 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-24T23:09:13.703 INFO:teuthology.orchestra.run.smithi191.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-24T23:09:13.703 INFO:teuthology.lock.ops:Updating smithi191.front.sepia.ceph.com on lock server 2023-03-24T23:09:13.727 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2023-03-24T23:09:13.747 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2023-03-24T23:09:13.747 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2023-03-24T23:09:13.807 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="Red Hat Enterprise Linux" 2023-03-24T23:09:13.807 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="8.6 (Ootpa)" 2023-03-24T23:09:13.807 INFO:teuthology.orchestra.run.smithi196.stdout:ID="rhel" 2023-03-24T23:09:13.807 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="fedora" 2023-03-24T23:09:13.808 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="8.6" 2023-03-24T23:09:13.808 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el8" 2023-03-24T23:09:13.808 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="Red Hat Enterprise Linux 8.6 (Ootpa)" 2023-03-24T23:09:13.808 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2023-03-24T23:09:13.808 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:redhat:enterprise_linux:8::baseos" 2023-03-24T23:09:13.808 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://www.redhat.com/" 2023-03-24T23:09:13.808 INFO:teuthology.orchestra.run.smithi196.stdout:DOCUMENTATION_URL="https://access.redhat.com/documentation/red_hat_enterprise_linux/8/" 2023-03-24T23:09:13.809 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-03-24T23:09:13.809 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:09:13.809 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 8" 2023-03-24T23:09:13.809 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_BUGZILLA_PRODUCT_VERSION=8.6 2023-03-24T23:09:13.809 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" 2023-03-24T23:09:13.809 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8.6" 2023-03-24T23:09:13.810 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2023-03-24T23:09:13.834 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-03-24T23:09:13.845 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-03-24T23:09:13.874 INFO:teuthology.task.internal:Checking for old test directory... 2023-03-24T23:09:13.874 DEBUG:teuthology.orchestra.run.smithi191:> test '!' -e /home/ubuntu/cephtest 2023-03-24T23:09:13.876 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2023-03-24T23:09:13.895 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-03-24T23:09:13.905 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-03-24T23:09:13.906 DEBUG:teuthology.orchestra.run.smithi191:> test -z $(ls -A /var/lib/ceph) 2023-03-24T23:09:13.932 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2023-03-24T23:09:13.960 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-03-24T23:09:14.023 INFO:teuthology.run_tasks:Running task kernel... 2023-03-24T23:09:14.039 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'} 2023-03-24T23:09:14.040 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}, 'client.2': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}, 'client.3': {'sha1': 'distro'}} 2023-03-24T23:09:14.040 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}, 'client.1': {'sha1': 'distro'}}, timeout 300 2023-03-24T23:09:14.041 DEBUG:teuthology.orchestra.run.smithi191:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-24T23:09:14.041 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2023-03-24T23:09:14.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-24T23:09:14.056 DEBUG:teuthology.orchestra.run.smithi191:> uname -r 2023-03-24T23:09:14.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-24T23:09:14.059 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2023-03-24T23:09:14.111 INFO:teuthology.orchestra.run.smithi191.stdout:4.18.0-372.9.1.el8.x86_64 2023-03-24T23:09:14.112 INFO:teuthology.task.kernel:Running kernel on smithi191: 4.18.0-372.9.1.el8.x86_64 2023-03-24T23:09:14.112 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum install -y kernel 2023-03-24T23:09:14.117 INFO:teuthology.orchestra.run.smithi196.stdout:4.18.0-372.9.1.el8.x86_64 2023-03-24T23:09:14.117 INFO:teuthology.task.kernel:Running kernel on smithi196: 4.18.0-372.9.1.el8.x86_64 2023-03-24T23:09:14.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2023-03-24T23:09:15.311 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-03-24T23:09:15.311 INFO:teuthology.orchestra.run.smithi196.stdout:Unable to read consumer identity 2023-03-24T23:09:15.595 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-03-24T23:09:15.595 INFO:teuthology.orchestra.run.smithi191.stdout:Unable to read consumer identity 2023-03-24T23:09:16.758 INFO:teuthology.orchestra.run.smithi196.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-03-24T23:09:17.469 INFO:teuthology.orchestra.run.smithi191.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-03-24T23:09:17.917 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 12 MB/s | 14 MB 00:01 2023-03-24T23:09:18.636 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 12 MB/s | 14 MB 00:01 2023-03-24T23:09:21.134 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 125 kB/s | 24 kB 00:00 2023-03-24T23:09:21.720 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 379 kB/s | 24 kB 00:00 2023-03-24T23:09:21.803 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-03-24T23:09:21.856 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-03-24T23:09:21.857 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2023-03-24T23:09:21.857 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-03-24T23:09:21.976 DEBUG:teuthology.orchestra.run.smithi196:> echo no | sudo yum reinstall kernel || true 2023-03-24T23:09:22.306 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-03-24T23:09:22.307 INFO:teuthology.orchestra.run.smithi196.stdout:Unable to read consumer identity 2023-03-24T23:09:22.365 INFO:teuthology.orchestra.run.smithi191.stdout:Package kernel-4.18.0-372.9.1.el8.x86_64 is already installed. 2023-03-24T23:09:22.416 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-03-24T23:09:22.417 INFO:teuthology.orchestra.run.smithi191.stdout:Nothing to do. 2023-03-24T23:09:22.417 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-03-24T23:09:22.533 DEBUG:teuthology.orchestra.run.smithi191:> echo no | sudo yum reinstall kernel || true 2023-03-24T23:09:22.565 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:02 ago on Fri 24 Mar 2023 11:09:20 PM UTC. 2023-03-24T23:09:22.608 INFO:teuthology.orchestra.run.smithi196.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-24T23:09:22.651 INFO:teuthology.orchestra.run.smithi196.stderr:Error: No packages marked for reinstall. 2023-03-24T23:09:22.699 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum reinstall -y kernel || true 2023-03-24T23:09:22.903 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-03-24T23:09:22.903 INFO:teuthology.orchestra.run.smithi191.stdout:Unable to read consumer identity 2023-03-24T23:09:23.027 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-03-24T23:09:23.027 INFO:teuthology.orchestra.run.smithi196.stdout:Unable to read consumer identity 2023-03-24T23:09:23.160 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:02 ago on Fri 24 Mar 2023 11:09:21 PM UTC. 2023-03-24T23:09:23.226 INFO:teuthology.orchestra.run.smithi191.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-24T23:09:23.264 INFO:teuthology.orchestra.run.smithi191.stderr:Error: No packages marked for reinstall. 2023-03-24T23:09:23.278 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:03 ago on Fri 24 Mar 2023 11:09:20 PM UTC. 2023-03-24T23:09:23.310 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum reinstall -y kernel || true 2023-03-24T23:09:23.317 INFO:teuthology.orchestra.run.smithi196.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-24T23:09:23.356 INFO:teuthology.orchestra.run.smithi196.stderr:Error: No packages marked for reinstall. 2023-03-24T23:09:23.402 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel --last | head -n 1 2023-03-24T23:09:23.481 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-03-24T23:09:23.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-03-24T23:09:23.483 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-03-24T23:09:23.483 DEBUG:teuthology.parallel:result is None 2023-03-24T23:09:23.639 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-03-24T23:09:23.640 INFO:teuthology.orchestra.run.smithi191.stdout:Unable to read consumer identity 2023-03-24T23:09:23.892 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:00:03 ago on Fri 24 Mar 2023 11:09:21 PM UTC. 2023-03-24T23:09:23.934 INFO:teuthology.orchestra.run.smithi191.stdout:Installed package kernel-4.18.0-372.9.1.el8.x86_64 (from anaconda) not available. 2023-03-24T23:09:23.965 INFO:teuthology.orchestra.run.smithi191.stderr:Error: No packages marked for reinstall. 2023-03-24T23:09:24.014 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q kernel --last | head -n 1 2023-03-24T23:09:24.094 INFO:teuthology.orchestra.run.smithi191.stdout:kernel-4.18.0-372.9.1.el8.x86_64 Thu 14 Jul 2022 06:23:21 PM UTC 2023-03-24T23:09:24.096 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-372.9.1.el8.x86_64 2023-03-24T23:09:24.096 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2023-03-24T23:09:24.096 DEBUG:teuthology.parallel:result is None 2023-03-24T23:09:24.096 INFO:teuthology.run_tasks:Running task internal.base... 2023-03-24T23:09:24.109 INFO:teuthology.task.internal:Creating test directory... 2023-03-24T23:09:24.110 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-24T23:09:24.112 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-03-24T23:09:24.131 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-03-24T23:09:24.150 INFO:teuthology.run_tasks:Running task internal.archive... 2023-03-24T23:09:24.183 INFO:teuthology.task.internal:Creating archive directory... 2023-03-24T23:09:24.184 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-24T23:09:24.187 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-03-24T23:09:24.226 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-03-24T23:09:24.233 INFO:teuthology.task.internal:Enabling coredump saving... 2023-03-24T23:09:24.234 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-24T23:09:24.264 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-03-24T23:09:24.303 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-24T23:09:24.303 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-24T23:09:24.314 INFO:teuthology.orchestra.run.smithi191.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-24T23:09:24.315 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-03-24T23:09:24.316 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-03-24T23:09:24.324 INFO:teuthology.task.internal:Configuring sudo... 2023-03-24T23:09:24.324 DEBUG:teuthology.orchestra.run.smithi191:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-24T23:09:24.358 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-03-24T23:09:24.385 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-03-24T23:09:24.395 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-03-24T23:09:24.395 DEBUG:teuthology.orchestra.run.smithi191:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-24T23:09:24.424 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-03-24T23:09:24.442 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-24T23:09:24.526 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-24T23:09:24.574 DEBUG:teuthology.orchestra.run.smithi191:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-24T23:09:24.663 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-24T23:09:24.699 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:09:24.700 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-24T23:09:24.771 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-24T23:09:24.819 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-03-24T23:09:24.863 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-24T23:09:24.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-03-24T23:09:24.988 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:09:24.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-03-24T23:09:25.056 DEBUG:teuthology.orchestra.run.smithi191:> sudo service rsyslog restart 2023-03-24T23:09:25.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2023-03-24T23:09:25.110 INFO:teuthology.orchestra.run.smithi191.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-24T23:09:25.152 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-03-24T23:09:25.493 INFO:teuthology.run_tasks:Running task internal.timer... 2023-03-24T23:09:25.501 INFO:teuthology.task.internal:Starting timer... 2023-03-24T23:09:25.501 INFO:teuthology.run_tasks:Running task pcp... 2023-03-24T23:09:25.557 INFO:teuthology.run_tasks:Running task selinux... 2023-03-24T23:09:25.577 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']} 2023-03-24T23:09:25.578 DEBUG:teuthology.orchestra.run.smithi191:> sudo service auditd rotate 2023-03-24T23:09:25.642 INFO:teuthology.orchestra.run.smithi191.stdout:Rotating logs: 2023-03-24T23:09:25.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2023-03-24T23:09:25.701 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2023-03-24T23:09:25.703 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-03-24T23:09:25.703 DEBUG:teuthology.orchestra.run.smithi191:> /usr/sbin/getenforce 2023-03-24T23:09:25.720 INFO:teuthology.orchestra.run.smithi191.stdout:Permissive 2023-03-24T23:09:25.721 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2023-03-24T23:09:25.739 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2023-03-24T23:09:25.739 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi191.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2023-03-24T23:09:25.739 DEBUG:teuthology.orchestra.run.smithi191:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-24T23:09:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-24T23:09:25.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-03-24T23:09:25.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-03-24T23:09:25.816 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-03-24T23:09:25.817 DEBUG:teuthology.orchestra.run.smithi191:> sudo /usr/sbin/setenforce permissive 2023-03-24T23:09:25.858 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2023-03-24T23:09:25.885 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-03-24T23:09:25.895 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-03-24T23:09:25.901 INFO:teuthology.repo_utils:Fetching main from origin 2023-03-24T23:09:26.040 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-03-24T23:09:26.061 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-03-24T23:09:26.062 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi191.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-03-24T23:16:45.046 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi191.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2023-03-24T23:16:45.048 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi191.front.sepia.ceph.com' 2023-03-24T23:16:45.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi191.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-24T23:16:45.119 DEBUG:teuthology.orchestra.run.smithi191:> true 2023-03-24T23:16:45.191 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2023-03-24T23:16:45.192 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-03-24T23:16:45.261 DEBUG:teuthology.orchestra.run.smithi196:> true 2023-03-24T23:16:45.331 INFO:teuthology.run_tasks:Running task clock... 2023-03-24T23:16:45.348 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-03-24T23:16:45.349 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-24T23:16:45.349 DEBUG:teuthology.orchestra.run.smithi191:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-24T23:16:45.352 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-03-24T23:16:45.353 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-03-24T23:16:45.381 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-24T23:16:45.396 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-24T23:16:45.401 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-03-24T23:16:45.416 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-03-24T23:16:45.469 INFO:teuthology.orchestra.run.smithi191.stderr:sudo: ntpd: command not found 2023-03-24T23:16:45.481 INFO:teuthology.orchestra.run.smithi191.stdout:506 Cannot talk to daemon 2023-03-24T23:16:45.489 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2023-03-24T23:16:45.495 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-24T23:16:45.501 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2023-03-24T23:16:45.509 INFO:teuthology.orchestra.run.smithi191.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-24T23:16:45.515 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-03-24T23:16:45.529 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-03-24T23:16:45.575 INFO:teuthology.orchestra.run.smithi191.stderr:bash: ntpq: command not found 2023-03-24T23:16:45.579 INFO:teuthology.orchestra.run.smithi191.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-24T23:16:45.579 INFO:teuthology.orchestra.run.smithi191.stdout:=============================================================================== 2023-03-24T23:16:45.579 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-24T23:16:45.580 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-24T23:16:45.580 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-24T23:16:45.580 INFO:teuthology.orchestra.run.smithi191.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-24T23:16:45.597 INFO:teuthology.orchestra.run.smithi196.stderr:bash: ntpq: command not found 2023-03-24T23:16:45.600 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-03-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2023-03-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-24T23:16:45.601 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-03-24T23:16:45.602 INFO:teuthology.run_tasks:Running task install... 2023-03-24T23:16:45.612 DEBUG:teuthology.task.install:project ceph 2023-03-24T23:16:45.613 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'}} 2023-03-24T23:16:45.613 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'} 2023-03-24T23:16:45.613 INFO:teuthology.task.install:Using flavor: default 2023-03-24T23:16:45.620 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-03-24T23:16:45.620 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-03-24T23:16:45.621 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb', 'tag': None, 'wait_for_package': False} 2023-03-24T23:16:45.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-24T23:16:45.624 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb', 'tag': None, 'wait_for_package': False} 2023-03-24T23:16:45.624 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-24T23:16:45.771 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/ 2023-03-24T23:16:45.771 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3124.gc19e8f2a 2023-03-24T23:16:45.776 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/ 2023-03-24T23:16:45.776 INFO:teuthology.task.install.rpm:Package version is 18.0.0-3124.gc19e8f2a 2023-03-24T23:16:45.878 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-24T23:16:45.878 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:16:45.879 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-24T23:16:45.888 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20230323.162417/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-03-24T23:16:45.888 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:16:45.888 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-03-24T23:16:45.907 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-24T23:16:45.907 DEBUG:teuthology.orchestra.run.smithi191:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-24T23:16:45.915 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-03-24T23:16:45.916 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c19e8f2a497e95f75abc43ec14bc6095e30dd2cb/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-03-24T23:16:45.974 DEBUG:teuthology.orchestra.run.smithi191:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-24T23:16:45.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-03-24T23:16:46.050 DEBUG:teuthology.orchestra.run.smithi191:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-24T23:16:46.059 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-03-24T23:16:46.116 INFO:teuthology.orchestra.run.smithi191.stdout:check_obsoletes = 1 2023-03-24T23:16:46.118 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum clean all 2023-03-24T23:16:46.125 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2023-03-24T23:16:46.126 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2023-03-24T23:16:46.490 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-03-24T23:16:46.497 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-03-24T23:16:46.726 INFO:teuthology.orchestra.run.smithi191.stdout:44 files removed 2023-03-24T23:16:46.743 INFO:teuthology.orchestra.run.smithi196.stdout:44 files removed 2023-03-24T23:16:46.774 DEBUG:teuthology.orchestra.run.smithi191:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-24T23:16:46.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-03-24T23:16:47.104 INFO:teuthology.orchestra.run.smithi191.stdout:Updating Subscription Management repositories. 2023-03-24T23:16:47.119 INFO:teuthology.orchestra.run.smithi196.stdout:Updating Subscription Management repositories. 2023-03-24T23:16:47.772 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 315 kB/s | 80 kB 00:00 2023-03-24T23:16:47.779 INFO:teuthology.orchestra.run.smithi191.stdout:ceph packages for x86_64 317 kB/s | 80 kB 00:00 2023-03-24T23:16:48.015 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 45 kB/s | 9.4 kB 00:00 2023-03-24T23:16:48.046 INFO:teuthology.orchestra.run.smithi191.stdout:ceph noarch packages 40 kB/s | 9.4 kB 00:00 2023-03-24T23:16:48.274 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 6.2 kB/s | 1.5 kB 00:00 2023-03-24T23:16:48.288 INFO:teuthology.orchestra.run.smithi191.stdout:ceph source packages 6.7 kB/s | 1.5 kB 00:00 2023-03-24T23:16:48.474 INFO:teuthology.orchestra.run.smithi196.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-03-24T23:16:48.498 INFO:teuthology.orchestra.run.smithi191.stdout:Copr repo for python3-asyncssh owned by ceph 18 kB/s | 3.5 kB 00:00 2023-03-24T23:16:49.525 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 13 MB/s | 14 MB 00:01 2023-03-24T23:16:49.573 INFO:teuthology.orchestra.run.smithi191.stdout:Extra Packages for Enterprise Linux 13 MB/s | 14 MB 00:01 2023-03-24T23:16:52.583 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 393 kB/s | 24 kB 00:00 2023-03-24T23:16:52.623 INFO:teuthology.orchestra.run.smithi191.stdout:lab-extras 393 kB/s | 24 kB 00:00 2023-03-24T23:16:53.419 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 62 MB/s | 47 MB 00:00 2023-03-24T23:16:53.583 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 59 MB/s | 53 MB 00:00 2023-03-24T23:17:01.319 INFO:teuthology.orchestra.run.smithi196.stdout:Red Hat Enterprise Linux 8 for x86_64 - BaseOS 63 MB/s | 53 MB 00:00 2023-03-24T23:17:01.760 INFO:teuthology.orchestra.run.smithi191.stdout:Red Hat Enterprise Linux 8 for x86_64 - AppStre 52 MB/s | 47 MB 00:00 2023-03-24T23:17:12.389 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-24T23:17:12.389 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-24T23:17:12.403 INFO:teuthology.orchestra.run.smithi196.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-24T23:17:12.404 INFO:teuthology.orchestra.run.smithi196.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-24T23:17:12.405 INFO:teuthology.orchestra.run.smithi196.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-24T23:17:12.407 INFO:teuthology.orchestra.run.smithi196.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-24T23:17:12.409 INFO:teuthology.orchestra.run.smithi196.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-24T23:17:12.412 INFO:teuthology.orchestra.run.smithi196.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-24T23:17:12.413 INFO:teuthology.orchestra.run.smithi196.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-24T23:17:12.540 INFO:teuthology.orchestra.run.smithi191.stdout:Package librados2-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-24T23:17:12.541 INFO:teuthology.orchestra.run.smithi191.stdout:Package librbd1-1:12.2.7-9.el8.x86_64 is already installed. 2023-03-24T23:17:12.545 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2023-03-24T23:17:12.555 INFO:teuthology.orchestra.run.smithi191.stdout:Package NetworkManager-1:1.36.0-4.el8.x86_64 is already installed. 2023-03-24T23:17:12.556 INFO:teuthology.orchestra.run.smithi191.stdout:Package iproute-5.15.0-4.el8.x86_64 is already installed. 2023-03-24T23:17:12.557 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================================================== 2023-03-24T23:17:12.557 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2023-03-24T23:17:12.557 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================================================== 2023-03-24T23:17:12.557 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2023-03-24T23:17:12.557 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-24T23:17:12.558 INFO:teuthology.orchestra.run.smithi196.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-24T23:17:12.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 6.4 k 2023-03-24T23:17:12.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 5.4 M 2023-03-24T23:17:12.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 846 k 2023-03-24T23:17:12.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 165 k 2023-03-24T23:17:12.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 1.5 M 2023-03-24T23:17:12.558 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 123 k 2023-03-24T23:17:12.559 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 1.7 M 2023-03-24T23:17:12.559 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 7.4 M 2023-03-24T23:17:12.559 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 47 k 2023-03-24T23:17:12.559 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 13 M 2023-03-24T23:17:12.559 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 41 M 2023-03-24T23:17:12.559 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 263 k 2023-03-24T23:17:12.559 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 83 k 2023-03-24T23:17:12.559 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-mirror x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 258 k 2023-03-24T23:17:12.560 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-top noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 18 k 2023-03-24T23:17:12.560 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-24T23:17:12.560 INFO:teuthology.orchestra.run.smithi196.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-24T23:17:12.560 INFO:teuthology.orchestra.run.smithi196.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-24T23:17:12.560 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 30 k 2023-03-24T23:17:12.560 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 769 k 2023-03-24T23:17:12.560 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 139 k 2023-03-24T23:17:12.561 INFO:teuthology.orchestra.run.smithi196.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-24T23:17:12.561 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-24T23:17:12.561 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-24T23:17:12.561 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-24T23:17:12.561 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-24T23:17:12.561 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-24T23:17:12.561 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 190 k 2023-03-24T23:17:12.561 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 370 k 2023-03-24T23:17:12.562 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 364 k 2023-03-24T23:17:12.562 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 107 k 2023-03-24T23:17:12.562 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 94 k 2023-03-24T23:17:12.562 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.4 M 2023-03-24T23:17:12.562 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 179 k 2023-03-24T23:17:12.562 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-24T23:17:12.562 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2023-03-24T23:17:12.563 INFO:teuthology.orchestra.run.smithi196.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-24T23:17:12.563 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.7 M 2023-03-24T23:17:12.563 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.6 M 2023-03-24T23:17:12.563 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2023-03-24T23:17:12.563 INFO:teuthology.orchestra.run.smithi196.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-24T23:17:12.563 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 22 M 2023-03-24T23:17:12.563 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 23 k 2023-03-24T23:17:12.563 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 2.3 M 2023-03-24T23:17:12.564 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 239 k 2023-03-24T23:17:12.564 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 4.6 M 2023-03-24T23:17:12.564 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 18 M 2023-03-24T23:17:12.564 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 15 k 2023-03-24T23:17:12.564 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 24 k 2023-03-24T23:17:12.564 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-24T23:17:12.564 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-24T23:17:12.564 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-24T23:17:12.565 INFO:teuthology.orchestra.run.smithi196.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-24T23:17:12.565 INFO:teuthology.orchestra.run.smithi196.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-24T23:17:12.565 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 137 k 2023-03-24T23:17:12.565 INFO:teuthology.orchestra.run.smithi196.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-24T23:17:12.565 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-24T23:17:12.565 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-24T23:17:12.565 INFO:teuthology.orchestra.run.smithi196.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-24T23:17:12.566 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-24T23:17:12.566 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-24T23:17:12.566 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 474 k 2023-03-24T23:17:12.566 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-24T23:17:12.566 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 6.5 M 2023-03-24T23:17:12.566 INFO:teuthology.orchestra.run.smithi196.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-24T23:17:12.566 INFO:teuthology.orchestra.run.smithi196.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-24T23:17:12.567 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-24T23:17:12.567 INFO:teuthology.orchestra.run.smithi196.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-24T23:17:12.567 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-24T23:17:12.567 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-24T23:17:12.567 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-24T23:17:12.567 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-24T23:17:12.567 INFO:teuthology.orchestra.run.smithi196.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-24T23:17:12.567 INFO:teuthology.orchestra.run.smithi196.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-24T23:17:12.568 INFO:teuthology.orchestra.run.smithi196.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-24T23:17:12.568 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-24T23:17:12.568 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-24T23:17:12.568 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-24T23:17:12.568 INFO:teuthology.orchestra.run.smithi196.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-24T23:17:12.568 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-24T23:17:12.568 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 45 k 2023-03-24T23:17:12.568 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 117 k 2023-03-24T23:17:12.569 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-24T23:17:12.569 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-24T23:17:12.569 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-24T23:17:12.569 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-24T23:17:12.569 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-24T23:17:12.569 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-24T23:17:12.569 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-24T23:17:12.570 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-24T23:17:12.570 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-24T23:17:12.570 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-24T23:17:12.570 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-24T23:17:12.570 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-24T23:17:12.570 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-24T23:17:12.570 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-24T23:17:12.570 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-24T23:17:12.571 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-24T23:17:12.571 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-24T23:17:12.571 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-24T23:17:12.571 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-24T23:17:12.571 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-24T23:17:12.571 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-24T23:17:12.571 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-24T23:17:12.571 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-24T23:17:12.572 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-24T23:17:12.572 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-24T23:17:12.572 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-24T23:17:12.572 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-24T23:17:12.572 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-24T23:17:12.572 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-24T23:17:12.572 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-24T23:17:12.572 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-24T23:17:12.573 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-24T23:17:12.573 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-24T23:17:12.573 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-24T23:17:12.573 INFO:teuthology.orchestra.run.smithi196.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-24T23:17:12.573 INFO:teuthology.orchestra.run.smithi196.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-24T23:17:12.573 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-24T23:17:12.573 INFO:teuthology.orchestra.run.smithi196.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-24T23:17:12.574 INFO:teuthology.orchestra.run.smithi196.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-24T23:17:12.574 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-24T23:17:12.574 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-24T23:17:12.574 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-24T23:17:12.574 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-24T23:17:12.574 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-24T23:17:12.574 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-24T23:17:12.575 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-24T23:17:12.575 INFO:teuthology.orchestra.run.smithi196.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-24T23:17:12.575 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-24T23:17:12.575 INFO:teuthology.orchestra.run.smithi196.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-24T23:17:12.575 INFO:teuthology.orchestra.run.smithi196.stdout:Enabling module streams: 2023-03-24T23:17:12.575 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql 10 2023-03-24T23:17:12.575 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:17:12.575 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2023-03-24T23:17:12.576 INFO:teuthology.orchestra.run.smithi196.stdout:=========================================================================================================================== 2023-03-24T23:17:12.576 INFO:teuthology.orchestra.run.smithi196.stdout:Install 128 Packages 2023-03-24T23:17:12.576 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 3 Packages 2023-03-24T23:17:12.576 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:17:12.577 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 197 M 2023-03-24T23:17:12.577 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2023-03-24T23:17:12.577 INFO:teuthology.orchestra.run.smithi191.stdout:Package util-linux-2.32.1-35.el8.x86_64 is already installed. 2023-03-24T23:17:12.578 INFO:teuthology.orchestra.run.smithi191.stdout:Package libacl-devel-2.2.53-1.el8.x86_64 is already installed. 2023-03-24T23:17:12.578 INFO:teuthology.orchestra.run.smithi191.stdout:Package libattr-devel-2.4.48-3.el8.x86_64 is already installed. 2023-03-24T23:17:12.578 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsdump-3.1.8-4.el8.x86_64 is already installed. 2023-03-24T23:17:12.578 INFO:teuthology.orchestra.run.smithi191.stdout:Package xfsprogs-5.0.0-10.el8.x86_64 is already installed. 2023-03-24T23:17:12.694 INFO:teuthology.orchestra.run.smithi191.stdout:Dependencies resolved. 2023-03-24T23:17:12.702 INFO:teuthology.orchestra.run.smithi196.stdout:(1/131): ceph-18.0.0-3124.gc19e8f2a.el8.x86_64. 50 kB/s | 6.4 kB 00:00 2023-03-24T23:17:12.706 INFO:teuthology.orchestra.run.smithi191.stdout:=========================================================================================================================== 2023-03-24T23:17:12.706 INFO:teuthology.orchestra.run.smithi191.stdout: Package Arch Version Repository Size 2023-03-24T23:17:12.706 INFO:teuthology.orchestra.run.smithi191.stdout:=========================================================================================================================== 2023-03-24T23:17:12.707 INFO:teuthology.orchestra.run.smithi191.stdout:Installing: 2023-03-24T23:17:12.707 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-initscripts-updown noarch 1:1.36.0-9.el8_6 rhel-8-for-x86_64-baseos-rpms 138 k 2023-03-24T23:17:12.707 INFO:teuthology.orchestra.run.smithi191.stdout: bison x86_64 3.0.4-10.el8 rhel-8-for-x86_64-appstream-rpms 688 k 2023-03-24T23:17:12.707 INFO:teuthology.orchestra.run.smithi191.stdout: ceph x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 6.4 k 2023-03-24T23:17:12.707 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 5.4 M 2023-03-24T23:17:12.707 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 846 k 2023-03-24T23:17:12.707 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 165 k 2023-03-24T23:17:12.708 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 1.5 M 2023-03-24T23:17:12.708 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 123 k 2023-03-24T23:17:12.708 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 1.7 M 2023-03-24T23:17:12.708 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 7.4 M 2023-03-24T23:17:12.708 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 47 k 2023-03-24T23:17:12.708 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 13 M 2023-03-24T23:17:12.708 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 41 M 2023-03-24T23:17:12.709 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 263 k 2023-03-24T23:17:12.709 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 83 k 2023-03-24T23:17:12.709 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 258 k 2023-03-24T23:17:12.709 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 18 k 2023-03-24T23:17:12.709 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel x86_64 0.186-1.el8 rhel-8-for-x86_64-baseos-rpms 60 k 2023-03-24T23:17:12.709 INFO:teuthology.orchestra.run.smithi191.stdout: flex x86_64 2.6.1-9.el8 rhel-8-for-x86_64-appstream-rpms 320 k 2023-03-24T23:17:12.709 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel x86_64 0.3.112-1.el8 rhel-8-for-x86_64-baseos-rpms 19 k 2023-03-24T23:17:12.710 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 30 k 2023-03-24T23:17:12.710 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 769 k 2023-03-24T23:17:12.710 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 139 k 2023-03-24T23:17:12.710 INFO:teuthology.orchestra.run.smithi191.stdout: libtool x86_64 2.4.6-25.el8 rhel-8-for-x86_64-appstream-rpms 709 k 2023-03-24T23:17:12.710 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel x86_64 2.32.1-35.el8 rhel-8-for-x86_64-baseos-rpms 98 k 2023-03-24T23:17:12.710 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel x86_64 1:1.1.1k-7.el8_6 rhel-8-for-x86_64-baseos-rpms 2.3 M 2023-03-24T23:17:12.710 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 1.5 M 2023-03-24T23:17:12.710 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 808 k 2023-03-24T23:17:12.711 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server x86_64 10.21-2.module+el8.6.0+15342+53518fac rhel-8-for-x86_64-appstream-rpms 5.1 M 2023-03-24T23:17:12.711 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 190 k 2023-03-24T23:17:12.711 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 370 k 2023-03-24T23:17:12.711 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 364 k 2023-03-24T23:17:12.711 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 107 k 2023-03-24T23:17:12.711 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 94 k 2023-03-24T23:17:12.711 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.4 M 2023-03-24T23:17:12.712 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 179 k 2023-03-24T23:17:12.712 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel x86_64 5.0.0-10.el8 rhel-8-for-x86_64-baseos-rpms 72 k 2023-03-24T23:17:12.712 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrading: 2023-03-24T23:17:12.712 INFO:teuthology.orchestra.run.smithi191.stdout: iproute x86_64 5.15.0-4.el8_6.1 rhel-8-for-x86_64-baseos-rpms 798 k 2023-03-24T23:17:12.712 INFO:teuthology.orchestra.run.smithi191.stdout: librados2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.7 M 2023-03-24T23:17:12.712 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 3.6 M 2023-03-24T23:17:12.712 INFO:teuthology.orchestra.run.smithi191.stdout:Installing dependencies: 2023-03-24T23:17:12.713 INFO:teuthology.orchestra.run.smithi191.stdout: automake noarch 1.16.1-7.el8 rhel-8-for-x86_64-appstream-rpms 713 k 2023-03-24T23:17:12.713 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 22 M 2023-03-24T23:17:12.713 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 23 k 2023-03-24T23:17:12.713 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 2.3 M 2023-03-24T23:17:12.713 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 239 k 2023-03-24T23:17:12.713 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 4.6 M 2023-03-24T23:17:12.713 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 18 M 2023-03-24T23:17:12.714 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts noarch 2:18.0.0-3124.gc19e8f2a.el8 ceph-noarch 15 k 2023-03-24T23:17:12.714 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 24 k 2023-03-24T23:17:12.714 INFO:teuthology.orchestra.run.smithi191.stdout: fuse x86_64 2.9.7-15.el8 rhel-8-for-x86_64-baseos-rpms 83 k 2023-03-24T23:17:12.714 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-03-24T23:17:12.714 INFO:teuthology.orchestra.run.smithi191.stdout: jq x86_64 1.6-3.el8 rhel-8-for-x86_64-appstream-rpms 202 k 2023-03-24T23:17:12.714 INFO:teuthology.orchestra.run.smithi191.stdout: keyutils-libs-devel x86_64 1.5.10-9.el8 rhel-8-for-x86_64-baseos-rpms 48 k 2023-03-24T23:17:12.714 INFO:teuthology.orchestra.run.smithi191.stdout: krb5-devel x86_64 1.18.2-14.el8 rhel-8-for-x86_64-baseos-rpms 560 k 2023-03-24T23:17:12.714 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 137 k 2023-03-24T23:17:12.715 INFO:teuthology.orchestra.run.smithi191.stdout: libcom_err-devel x86_64 1.45.6-4.el8 rhel-8-for-x86_64-baseos-rpms 39 k 2023-03-24T23:17:12.715 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 644 k 2023-03-24T23:17:12.715 INFO:teuthology.orchestra.run.smithi191.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-03-24T23:17:12.715 INFO:teuthology.orchestra.run.smithi191.stdout: libpq x86_64 13.5-1.el8 rhel-8-for-x86_64-appstream-rpms 198 k 2023-03-24T23:17:12.715 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath x86_64 8.5.0-10.1.el8_6 rhel-8-for-x86_64-baseos-rpms 171 k 2023-03-24T23:17:12.715 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq x86_64 0.9.0-3.el8 rhel-8-for-x86_64-baseos-rpms 47 k 2023-03-24T23:17:12.715 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 474 k 2023-03-24T23:17:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka x86_64 0.11.4-3.el8 rhel-8-for-x86_64-appstream-rpms 354 k 2023-03-24T23:17:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2 x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 6.5 M 2023-03-24T23:17:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: libselinux-devel x86_64 2.9-5.el8 rhel-8-for-x86_64-baseos-rpms 200 k 2023-03-24T23:17:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: libsepol-devel x86_64 2.9-3.el8 rhel-8-for-x86_64-baseos-rpms 87 k 2023-03-24T23:17:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-03-24T23:17:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: libverto-devel x86_64 0.3.0-5.el8 rhel-8-for-x86_64-baseos-rpms 18 k 2023-03-24T23:17:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust x86_64 2.8.1-11.el8 rhel-8-for-x86_64-appstream-rpms 259 k 2023-03-24T23:17:12.716 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma x86_64 6.8.2-2.el8 rhel-8-for-x86_64-appstream-rpms 187 k 2023-03-24T23:17:12.717 INFO:teuthology.orchestra.run.smithi191.stdout: openblas x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.7 M 2023-03-24T23:17:12.717 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads x86_64 0.3.15-3.el8 rhel-8-for-x86_64-appstream-rpms 4.9 M 2023-03-24T23:17:12.717 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-devel x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 605 k 2023-03-24T23:17:12.717 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf16 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 229 k 2023-03-24T23:17:12.717 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf32 x86_64 10.32-2.el8 rhel-8-for-x86_64-baseos-rpms 220 k 2023-03-24T23:17:12.717 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-03-24T23:17:12.717 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel noarch 2.5.1-7.el8 rhel-8-for-x86_64-appstream-rpms 4.8 M 2023-03-24T23:17:12.718 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-03-24T23:17:12.718 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-03-24T23:17:12.718 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-03-24T23:17:12.718 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 45 k 2023-03-24T23:17:12.718 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common x86_64 2:18.0.0-3124.gc19e8f2a.el8 ceph 117 k 2023-03-24T23:17:12.718 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-03-24T23:17:12.718 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi x86_64 1.11.5-5.el8 rhel-8-for-x86_64-baseos-rpms 238 k 2023-03-24T23:17:12.718 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-03-24T23:17:12.719 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-03-24T23:17:12.719 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography x86_64 3.2.1-5.el8 rhel-8-for-x86_64-baseos-rpms 559 k 2023-03-24T23:17:12.719 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-03-24T23:17:12.719 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-03-24T23:17:12.719 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-03-24T23:17:12.719 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2 noarch 2.10.1-3.el8 rhel-8-for-x86_64-appstream-rpms 538 k 2023-03-24T23:17:12.719 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch noarch 1.21-2.el8 rhel-8-for-x86_64-appstream-rpms 27 k 2023-03-24T23:17:12.719 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer noarch 1.10-11.el8 rhel-8-for-x86_64-appstream-rpms 20 k 2023-03-24T23:17:12.720 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt noarch 1.6.1-2.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-24T23:17:12.720 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-03-24T23:17:12.720 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-03-24T23:17:12.720 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako noarch 1.0.6-13.el8 rhel-8-for-x86_64-appstream-rpms 157 k 2023-03-24T23:17:12.720 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe x86_64 0.23-19.el8 rhel-8-for-x86_64-appstream-rpms 39 k 2023-03-24T23:17:12.720 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-03-24T23:17:12.720 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-03-24T23:17:12.721 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 3.7 M 2023-03-24T23:17:12.721 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 rhel-8-for-x86_64-appstream-rpms 225 k 2023-03-24T23:17:12.721 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib noarch 2.1.0-1.el8 rhel-8-for-x86_64-baseos-rpms 155 k 2023-03-24T23:17:12.721 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-03-24T23:17:12.721 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-03-24T23:17:12.721 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable noarch 0.7.2-14.el8 rhel-8-for-x86_64-appstream-rpms 44 k 2023-03-24T23:17:12.721 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 rhel-8-for-x86_64-appstream-rpms 103 k 2023-03-24T23:17:12.721 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1 noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 126 k 2023-03-24T23:17:12.722 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 rhel-8-for-x86_64-appstream-rpms 110 k 2023-03-24T23:17:12.722 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser noarch 2.14-14.el8 rhel-8-for-x86_64-baseos-rpms 109 k 2023-03-24T23:17:12.722 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz noarch 2017.2-9.el8 rhel-8-for-x86_64-appstream-rpms 54 k 2023-03-24T23:17:12.722 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-03-24T23:17:12.722 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 rhel-8-for-x86_64-baseos-rpms 43 k 2023-03-24T23:17:12.722 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-03-24T23:17:12.722 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa noarch 4.8-1.el8 epel 61 k 2023-03-24T23:17:12.722 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy x86_64 1.0.0-21.module+el8.5.0+10916+41bd434d rhel-8-for-x86_64-appstream-rpms 14 M 2023-03-24T23:17:12.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-03-24T23:17:12.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-03-24T23:17:12.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-03-24T23:17:12.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-03-24T23:17:12.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2023-03-24T23:17:12.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-03-24T23:17:12.723 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-03-24T23:17:12.724 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-03-24T23:17:12.724 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug noarch 0.12.2-4.el8 rhel-8-for-x86_64-appstream-rpms 457 k 2023-03-24T23:17:12.724 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-03-24T23:17:12.724 INFO:teuthology.orchestra.run.smithi191.stdout: socat x86_64 1.7.4.1-1.el8 rhel-8-for-x86_64-appstream-rpms 323 k 2023-03-24T23:17:12.724 INFO:teuthology.orchestra.run.smithi191.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-03-24T23:17:12.724 INFO:teuthology.orchestra.run.smithi191.stdout: uuid x86_64 1.6.2-43.el8 rhel-8-for-x86_64-appstream-rpms 64 k 2023-03-24T23:17:12.724 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet x86_64 1.6.1-20.el8 lab-extras 70 k 2023-03-24T23:17:12.724 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel x86_64 1.2.11-18.el8_5 rhel-8-for-x86_64-baseos-rpms 58 k 2023-03-24T23:17:12.725 INFO:teuthology.orchestra.run.smithi191.stdout:Enabling module streams: 2023-03-24T23:17:12.725 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql 10 2023-03-24T23:17:12.725 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:17:12.725 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction Summary 2023-03-24T23:17:12.725 INFO:teuthology.orchestra.run.smithi191.stdout:=========================================================================================================================== 2023-03-24T23:17:12.725 INFO:teuthology.orchestra.run.smithi191.stdout:Install 128 Packages 2023-03-24T23:17:12.725 INFO:teuthology.orchestra.run.smithi191.stdout:Upgrade 3 Packages 2023-03-24T23:17:12.726 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:17:12.726 INFO:teuthology.orchestra.run.smithi191.stdout:Total download size: 197 M 2023-03-24T23:17:12.726 INFO:teuthology.orchestra.run.smithi191.stdout:Downloading Packages: 2023-03-24T23:17:12.852 INFO:teuthology.orchestra.run.smithi191.stdout:(1/131): ceph-18.0.0-3124.gc19e8f2a.el8.x86_64. 50 kB/s | 6.4 kB 00:00 2023-03-24T23:17:12.902 INFO:teuthology.orchestra.run.smithi196.stdout:(2/131): ceph-fuse-18.0.0-3124.gc19e8f2a.el8.x8 4.1 MB/s | 846 kB 00:00 2023-03-24T23:17:12.960 INFO:teuthology.orchestra.run.smithi196.stdout:(3/131): ceph-immutable-object-cache-18.0.0-312 2.8 MB/s | 165 kB 00:00 2023-03-24T23:17:13.143 INFO:teuthology.orchestra.run.smithi191.stdout:(2/131): ceph-fuse-18.0.0-3124.gc19e8f2a.el8.x8 2.8 MB/s | 846 kB 00:00 2023-03-24T23:17:13.218 INFO:teuthology.orchestra.run.smithi191.stdout:(3/131): ceph-immutable-object-cache-18.0.0-312 2.2 MB/s | 165 kB 00:00 2023-03-24T23:17:13.444 INFO:teuthology.orchestra.run.smithi196.stdout:(4/131): ceph-base-18.0.0-3124.gc19e8f2a.el8.x8 6.2 MB/s | 5.4 MB 00:00 2023-03-24T23:17:13.519 INFO:teuthology.orchestra.run.smithi196.stdout:(5/131): ceph-mds-18.0.0-3124.gc19e8f2a.el8.x86 4.1 MB/s | 2.3 MB 00:00 2023-03-24T23:17:13.800 INFO:teuthology.orchestra.run.smithi196.stdout:(6/131): ceph-mgr-18.0.0-3124.gc19e8f2a.el8.x86 4.3 MB/s | 1.5 MB 00:00 2023-03-24T23:17:13.843 INFO:teuthology.orchestra.run.smithi191.stdout:(4/131): ceph-mds-18.0.0-3124.gc19e8f2a.el8.x86 3.6 MB/s | 2.3 MB 00:00 2023-03-24T23:17:14.552 INFO:teuthology.orchestra.run.smithi191.stdout:(5/131): ceph-mgr-18.0.0-3124.gc19e8f2a.el8.x86 2.2 MB/s | 1.5 MB 00:00 2023-03-24T23:17:14.603 INFO:teuthology.orchestra.run.smithi196.stdout:(7/131): ceph-mon-18.0.0-3124.gc19e8f2a.el8.x86 4.2 MB/s | 4.6 MB 00:01 2023-03-24T23:17:15.002 INFO:teuthology.orchestra.run.smithi191.stdout:(6/131): ceph-base-18.0.0-3124.gc19e8f2a.el8.x8 2.4 MB/s | 5.4 MB 00:02 2023-03-24T23:17:15.670 INFO:teuthology.orchestra.run.smithi196.stdout:(8/131): ceph-common-18.0.0-3124.gc19e8f2a.el8. 7.0 MB/s | 22 MB 00:03 2023-03-24T23:17:15.745 INFO:teuthology.orchestra.run.smithi196.stdout:(9/131): ceph-selinux-18.0.0-3124.gc19e8f2a.el8 325 kB/s | 24 kB 00:00 2023-03-24T23:17:16.078 INFO:teuthology.orchestra.run.smithi191.stdout:(7/131): ceph-mon-18.0.0-3124.gc19e8f2a.el8.x86 3.0 MB/s | 4.6 MB 00:01 2023-03-24T23:17:16.779 INFO:teuthology.orchestra.run.smithi196.stdout:(10/131): ceph-osd-18.0.0-3124.gc19e8f2a.el8.x8 5.9 MB/s | 18 MB 00:02 2023-03-24T23:17:16.803 INFO:teuthology.orchestra.run.smithi191.stdout:(8/131): ceph-common-18.0.0-3124.gc19e8f2a.el8. 5.3 MB/s | 22 MB 00:04 2023-03-24T23:17:16.846 INFO:teuthology.orchestra.run.smithi196.stdout:(11/131): cephfs-mirror-18.0.0-3124.gc19e8f2a.e 3.8 MB/s | 258 kB 00:00 2023-03-24T23:17:16.886 INFO:teuthology.orchestra.run.smithi191.stdout:(9/131): ceph-selinux-18.0.0-3124.gc19e8f2a.el8 292 kB/s | 24 kB 00:00 2023-03-24T23:17:16.946 INFO:teuthology.orchestra.run.smithi196.stdout:(12/131): libcephfs-devel-18.0.0-3124.gc19e8f2a 305 kB/s | 30 kB 00:00 2023-03-24T23:17:17.188 INFO:teuthology.orchestra.run.smithi196.stdout:(13/131): libcephfs2-18.0.0-3124.gc19e8f2a.el8. 3.1 MB/s | 769 kB 00:00 2023-03-24T23:17:17.322 INFO:teuthology.orchestra.run.smithi196.stdout:(14/131): ceph-radosgw-18.0.0-3124.gc19e8f2a.el 4.8 MB/s | 13 MB 00:02 2023-03-24T23:17:17.347 INFO:teuthology.orchestra.run.smithi196.stdout:(15/131): libcephsqlite-18.0.0-3124.gc19e8f2a.e 865 kB/s | 137 kB 00:00 2023-03-24T23:17:17.405 INFO:teuthology.orchestra.run.smithi196.stdout:(16/131): librados-devel-18.0.0-3124.gc19e8f2a. 1.6 MB/s | 139 kB 00:00 2023-03-24T23:17:17.497 INFO:teuthology.orchestra.run.smithi196.stdout:(17/131): libradosstriper1-18.0.0-3124.gc19e8f2 3.1 MB/s | 474 kB 00:00 2023-03-24T23:17:17.581 INFO:teuthology.orchestra.run.smithi196.stdout:(18/131): python3-ceph-argparse-18.0.0-3124.gc1 535 kB/s | 45 kB 00:00 2023-03-24T23:17:17.673 INFO:teuthology.orchestra.run.smithi196.stdout:(19/131): python3-ceph-common-18.0.0-3124.gc19e 1.2 MB/s | 117 kB 00:00 2023-03-24T23:17:17.773 INFO:teuthology.orchestra.run.smithi196.stdout:(20/131): python3-cephfs-18.0.0-3124.gc19e8f2a. 1.9 MB/s | 190 kB 00:00 2023-03-24T23:17:17.940 INFO:teuthology.orchestra.run.smithi196.stdout:(21/131): python3-rados-18.0.0-3124.gc19e8f2a.e 2.2 MB/s | 370 kB 00:00 2023-03-24T23:17:18.049 INFO:teuthology.orchestra.run.smithi196.stdout:(22/131): python3-rbd-18.0.0-3124.gc19e8f2a.el8 3.3 MB/s | 364 kB 00:00 2023-03-24T23:17:18.149 INFO:teuthology.orchestra.run.smithi196.stdout:(23/131): python3-rgw-18.0.0-3124.gc19e8f2a.el8 1.0 MB/s | 107 kB 00:00 2023-03-24T23:17:18.241 INFO:teuthology.orchestra.run.smithi196.stdout:(24/131): rbd-fuse-18.0.0-3124.gc19e8f2a.el8.x8 1.0 MB/s | 94 kB 00:00 2023-03-24T23:17:18.758 INFO:teuthology.orchestra.run.smithi196.stdout:(25/131): librgw2-18.0.0-3124.gc19e8f2a.el8.x86 4.8 MB/s | 6.5 MB 00:01 2023-03-24T23:17:18.858 INFO:teuthology.orchestra.run.smithi196.stdout:(26/131): rbd-nbd-18.0.0-3124.gc19e8f2a.el8.x86 1.8 MB/s | 179 kB 00:00 2023-03-24T23:17:18.950 INFO:teuthology.orchestra.run.smithi196.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3124.g 252 kB/s | 23 kB 00:00 2023-03-24T23:17:19.042 INFO:teuthology.orchestra.run.smithi196.stdout:(28/131): rbd-mirror-18.0.0-3124.gc19e8f2a.el8. 4.3 MB/s | 3.4 MB 00:00 2023-03-24T23:17:19.067 INFO:teuthology.orchestra.run.smithi196.stdout:(29/131): ceph-mgr-cephadm-18.0.0-3124.gc19e8f2 1.0 MB/s | 123 kB 00:00 2023-03-24T23:17:19.278 INFO:teuthology.orchestra.run.smithi191.stdout:(10/131): ceph-radosgw-18.0.0-3124.gc19e8f2a.el 4.1 MB/s | 13 MB 00:03 2023-03-24T23:17:19.395 INFO:teuthology.orchestra.run.smithi191.stdout:(11/131): cephfs-mirror-18.0.0-3124.gc19e8f2a.e 2.2 MB/s | 258 kB 00:00 2023-03-24T23:17:19.443 INFO:teuthology.orchestra.run.smithi196.stdout:(30/131): ceph-mgr-dashboard-18.0.0-3124.gc19e8 4.2 MB/s | 1.7 MB 00:00 2023-03-24T23:17:19.478 INFO:teuthology.orchestra.run.smithi191.stdout:(12/131): libcephfs-devel-18.0.0-3124.gc19e8f2a 365 kB/s | 30 kB 00:00 2023-03-24T23:17:19.543 INFO:teuthology.orchestra.run.smithi196.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3124.gc1 2.3 MB/s | 239 kB 00:00 2023-03-24T23:17:19.635 INFO:teuthology.orchestra.run.smithi196.stdout:(32/131): ceph-mgr-rook-18.0.0-3124.gc19e8f2a.e 518 kB/s | 47 kB 00:00 2023-03-24T23:17:19.735 INFO:teuthology.orchestra.run.smithi196.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3124.gc 145 kB/s | 15 kB 00:00 2023-03-24T23:17:19.737 INFO:teuthology.orchestra.run.smithi191.stdout:(13/131): libcephfs2-18.0.0-3124.gc19e8f2a.el8. 2.9 MB/s | 769 kB 00:00 2023-03-24T23:17:19.836 INFO:teuthology.orchestra.run.smithi196.stdout:(34/131): ceph-volume-18.0.0-3124.gc19e8f2a.el8 2.6 MB/s | 263 kB 00:00 2023-03-24T23:17:19.846 INFO:teuthology.orchestra.run.smithi191.stdout:(14/131): libcephsqlite-18.0.0-3124.gc19e8f2a.e 1.2 MB/s | 137 kB 00:00 2023-03-24T23:17:19.912 INFO:teuthology.orchestra.run.smithi196.stdout:(35/131): cephadm-18.0.0-3124.gc19e8f2a.el8.noa 1.1 MB/s | 83 kB 00:00 2023-03-24T23:17:19.946 INFO:teuthology.orchestra.run.smithi191.stdout:(15/131): librados-devel-18.0.0-3124.gc19e8f2a. 1.4 MB/s | 139 kB 00:00 2023-03-24T23:17:19.995 INFO:teuthology.orchestra.run.smithi196.stdout:(36/131): cephfs-top-18.0.0-3124.gc19e8f2a.el8. 216 kB/s | 18 kB 00:00 2023-03-24T23:17:20.121 INFO:teuthology.orchestra.run.smithi191.stdout:(16/131): libradosstriper1-18.0.0-3124.gc19e8f2 2.6 MB/s | 474 kB 00:00 2023-03-24T23:17:20.445 INFO:teuthology.orchestra.run.smithi196.stdout:(37/131): ceph-test-18.0.0-3124.gc19e8f2a.el8.x 8.6 MB/s | 41 MB 00:04 2023-03-24T23:17:20.537 INFO:teuthology.orchestra.run.smithi196.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 5.0 MB/s | 7.4 MB 00:01 2023-03-24T23:17:20.547 INFO:teuthology.orchestra.run.smithi191.stdout:(17/131): ceph-osd-18.0.0-3124.gc19e8f2a.el8.x8 3.2 MB/s | 18 MB 00:05 2023-03-24T23:17:20.562 INFO:teuthology.orchestra.run.smithi196.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 540 kB/s | 306 kB 00:00 2023-03-24T23:17:20.597 INFO:teuthology.orchestra.run.smithi191.stdout:(18/131): python3-ceph-argparse-18.0.0-3124.gc1 894 kB/s | 45 kB 00:00 2023-03-24T23:17:20.646 INFO:teuthology.orchestra.run.smithi196.stdout:(40/131): python3-asyncssh-2.7.0-2.el8.noarch.r 5.6 MB/s | 476 kB 00:00 2023-03-24T23:17:20.656 INFO:teuthology.orchestra.run.smithi191.stdout:(19/131): python3-ceph-common-18.0.0-3124.gc19e 2.0 MB/s | 117 kB 00:00 2023-03-24T23:17:20.688 INFO:teuthology.orchestra.run.smithi196.stdout:(41/131): liboath-2.6.2-3.el8.x86_64.rpm 243 kB/s | 59 kB 00:00 2023-03-24T23:17:20.713 INFO:teuthology.orchestra.run.smithi196.stdout:(42/131): libunwind-1.3.1-3.el8.x86_64.rpm 427 kB/s | 75 kB 00:00 2023-03-24T23:17:20.723 INFO:teuthology.orchestra.run.smithi191.stdout:(20/131): python3-cephfs-18.0.0-3124.gc19e8f2a. 2.8 MB/s | 190 kB 00:00 2023-03-24T23:17:20.738 INFO:teuthology.orchestra.run.smithi196.stdout:(43/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 474 kB/s | 44 kB 00:00 2023-03-24T23:17:20.788 INFO:teuthology.orchestra.run.smithi196.stdout:(44/131): python3-certifi-2018.10.15-7.el8.noar 327 kB/s | 16 kB 00:00 2023-03-24T23:17:20.814 INFO:teuthology.orchestra.run.smithi196.stdout:(45/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.4 MB/s | 185 kB 00:00 2023-03-24T23:17:20.830 INFO:teuthology.orchestra.run.smithi196.stdout:(46/131): python3-cachetools-3.1.1-4.el8.noarch 359 kB/s | 33 kB 00:00 2023-03-24T23:17:20.856 INFO:teuthology.orchestra.run.smithi191.stdout:(21/131): python3-rados-18.0.0-3124.gc19e8f2a.e 2.7 MB/s | 370 kB 00:00 2023-03-24T23:17:20.881 INFO:teuthology.orchestra.run.smithi196.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.8 MB/s | 173 kB 00:00 2023-03-24T23:17:20.897 INFO:teuthology.orchestra.run.smithi196.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 1.2 MB/s | 81 kB 00:00 2023-03-24T23:17:20.923 INFO:teuthology.orchestra.run.smithi196.stdout:(49/131): python3-cherrypy-18.4.0-1.el8.noarch. 4.1 MB/s | 384 kB 00:00 2023-03-24T23:17:20.940 INFO:teuthology.orchestra.run.smithi196.stdout:(50/131): python3-jaraco-6.2-6.el8.noarch.rpm 189 kB/s | 11 kB 00:00 2023-03-24T23:17:20.956 INFO:teuthology.orchestra.run.smithi196.stdout:(51/131): python3-jaraco-functools-2.0-4.el8.no 299 kB/s | 18 kB 00:00 2023-03-24T23:17:20.965 INFO:teuthology.orchestra.run.smithi191.stdout:(22/131): python3-rbd-18.0.0-3124.gc19e8f2a.el8 3.3 MB/s | 364 kB 00:00 2023-03-24T23:17:20.982 INFO:teuthology.orchestra.run.smithi196.stdout:(52/131): python3-logutils-0.3.5-11.el8.noarch. 1.1 MB/s | 49 kB 00:00 2023-03-24T23:17:20.999 INFO:teuthology.orchestra.run.smithi196.stdout:(53/131): python3-more-itertools-7.2.0-3.el8.no 1.4 MB/s | 59 kB 00:00 2023-03-24T23:17:21.024 INFO:teuthology.orchestra.run.smithi196.stdout:(54/131): python3-natsort-7.1.1-2.el8.noarch.rp 1.4 MB/s | 60 kB 00:00 2023-03-24T23:17:21.057 INFO:teuthology.orchestra.run.smithi191.stdout:(23/131): librgw2-18.0.0-3124.gc19e8f2a.el8.x86 7.0 MB/s | 6.5 MB 00:00 2023-03-24T23:17:21.074 INFO:teuthology.orchestra.run.smithi196.stdout:(55/131): python3-kubernetes-11.0.0-6.el8.noarc 7.7 MB/s | 1.2 MB 00:00 2023-03-24T23:17:21.082 INFO:teuthology.orchestra.run.smithi191.stdout:(24/131): python3-rgw-18.0.0-3124.gc19e8f2a.el8 912 kB/s | 107 kB 00:00 2023-03-24T23:17:21.091 INFO:teuthology.orchestra.run.smithi196.stdout:(56/131): python3-portend-2.6-1.el8.noarch.rpm 233 kB/s | 16 kB 00:00 2023-03-24T23:17:21.099 INFO:teuthology.orchestra.run.smithi191.stdout:(25/131): rbd-fuse-18.0.0-3124.gc19e8f2a.el8.x8 2.2 MB/s | 94 kB 00:00 2023-03-24T23:17:21.116 INFO:teuthology.orchestra.run.smithi196.stdout:(57/131): python3-pecan-1.3.2-9.el8.noarch.rpm 2.4 MB/s | 283 kB 00:00 2023-03-24T23:17:21.133 INFO:teuthology.orchestra.run.smithi196.stdout:(58/131): python3-repoze-lru-0.7-6.el8.noarch.r 574 kB/s | 34 kB 00:00 2023-03-24T23:17:21.141 INFO:teuthology.orchestra.run.smithi191.stdout:(26/131): rbd-nbd-18.0.0-3124.gc19e8f2a.el8.x86 4.2 MB/s | 179 kB 00:00 2023-03-24T23:17:21.159 INFO:teuthology.orchestra.run.smithi196.stdout:(59/131): python3-rsa-4.8-1.el8.noarch.rpm 1.4 MB/s | 61 kB 00:00 2023-03-24T23:17:21.184 INFO:teuthology.orchestra.run.smithi196.stdout:(60/131): python3-routes-2.4.1-12.el8.noarch.rp 2.1 MB/s | 196 kB 00:00 2023-03-24T23:17:21.200 INFO:teuthology.orchestra.run.smithi191.stdout:(27/131): ceph-grafana-dashboards-18.0.0-3124.g 396 kB/s | 23 kB 00:00 2023-03-24T23:17:21.201 INFO:teuthology.orchestra.run.smithi196.stdout:(61/131): python3-simplegeneric-0.8.1-17.el8.no 278 kB/s | 19 kB 00:00 2023-03-24T23:17:21.218 INFO:teuthology.orchestra.run.smithi196.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 407 kB/s | 24 kB 00:00 2023-03-24T23:17:21.235 INFO:teuthology.orchestra.run.smithi196.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 578 kB/s | 29 kB 00:00 2023-03-24T23:17:21.252 INFO:teuthology.orchestra.run.smithi196.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 545 kB/s | 27 kB 00:00 2023-03-24T23:17:21.259 INFO:teuthology.orchestra.run.smithi191.stdout:(28/131): ceph-mgr-cephadm-18.0.0-3124.gc19e8f2 2.1 MB/s | 123 kB 00:00 2023-03-24T23:17:21.285 INFO:teuthology.orchestra.run.smithi196.stdout:(65/131): python3-waitress-1.2.1-2.el8.1.noarch 3.5 MB/s | 241 kB 00:00 2023-03-24T23:17:21.311 INFO:teuthology.orchestra.run.smithi196.stdout:(66/131): python3-websocket-client-0.56.0-5.el8 1.0 MB/s | 61 kB 00:00 2023-03-24T23:17:21.336 INFO:teuthology.orchestra.run.smithi196.stdout:(67/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.4 MB/s | 251 kB 00:00 2023-03-24T23:17:21.361 INFO:teuthology.orchestra.run.smithi196.stdout:(68/131): python3-webtest-2.0.33-1.el8.noarch.r 1.1 MB/s | 85 kB 00:00 2023-03-24T23:17:21.387 INFO:teuthology.orchestra.run.smithi196.stdout:(69/131): python3-zc-lockfile-2.0-2.el8.noarch. 303 kB/s | 23 kB 00:00 2023-03-24T23:17:21.412 INFO:teuthology.orchestra.run.smithi196.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-03-24T23:17:21.479 INFO:teuthology.orchestra.run.smithi196.stdout:(71/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.8 MB/s | 110 kB 00:00 2023-03-24T23:17:21.501 INFO:teuthology.orchestra.run.smithi191.stdout:(29/131): ceph-mgr-dashboard-18.0.0-3124.gc19e8 7.0 MB/s | 1.7 MB 00:00 2023-03-24T23:17:21.521 INFO:teuthology.orchestra.run.smithi196.stdout:(72/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-03-24T23:17:21.571 INFO:teuthology.orchestra.run.smithi196.stdout:(73/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-03-24T23:17:21.613 INFO:teuthology.orchestra.run.smithi196.stdout:(74/131): thrift-0.13.0-2.el8.x86_64.rpm 6.3 MB/s | 1.7 MB 00:00 2023-03-24T23:17:21.647 INFO:teuthology.orchestra.run.smithi196.stdout:(75/131): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-03-24T23:17:21.659 INFO:teuthology.orchestra.run.smithi191.stdout:(30/131): rbd-mirror-18.0.0-3124.gc19e8f2a.el8. 6.0 MB/s | 3.4 MB 00:00 2023-03-24T23:17:21.688 INFO:teuthology.orchestra.run.smithi196.stdout:(76/131): python3-markupsafe-0.23-19.el8.x86_64 517 kB/s | 39 kB 00:00 2023-03-24T23:17:21.722 INFO:teuthology.orchestra.run.smithi196.stdout:(77/131): bison-3.0.4-10.el8.x86_64.rpm 9.7 MB/s | 688 kB 00:00 2023-03-24T23:17:21.755 INFO:teuthology.orchestra.run.smithi196.stdout:(78/131): python3-jinja2-2.10.1-3.el8.noarch.rp 7.9 MB/s | 538 kB 00:00 2023-03-24T23:17:21.759 INFO:teuthology.orchestra.run.smithi191.stdout:(31/131): ceph-mgr-modules-core-18.0.0-3124.gc1 2.3 MB/s | 239 kB 00:00 2023-03-24T23:17:21.781 INFO:teuthology.orchestra.run.smithi196.stdout:(79/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-03-24T23:17:21.818 INFO:teuthology.orchestra.run.smithi191.stdout:(32/131): ceph-mgr-rook-18.0.0-3124.gc19e8f2a.e 815 kB/s | 47 kB 00:00 2023-03-24T23:17:21.831 INFO:teuthology.orchestra.run.smithi196.stdout:(80/131): oniguruma-6.8.2-2.el8.x86_64.rpm 2.4 MB/s | 187 kB 00:00 2023-03-24T23:17:21.865 INFO:teuthology.orchestra.run.smithi196.stdout:(81/131): automake-1.16.1-7.el8.noarch.rpm 8.3 MB/s | 713 kB 00:00 2023-03-24T23:17:21.877 INFO:teuthology.orchestra.run.smithi191.stdout:(33/131): ceph-prometheus-alerts-18.0.0-3124.gc 249 kB/s | 15 kB 00:00 2023-03-24T23:17:21.906 INFO:teuthology.orchestra.run.smithi196.stdout:(82/131): python3-jsonpatch-1.21-2.el8.noarch.r 640 kB/s | 27 kB 00:00 2023-03-24T23:17:21.957 INFO:teuthology.orchestra.run.smithi196.stdout:(83/131): flex-2.6.1-9.el8.x86_64.rpm 6.3 MB/s | 320 kB 00:00 2023-03-24T23:17:21.985 INFO:teuthology.orchestra.run.smithi191.stdout:(34/131): ceph-volume-18.0.0-3124.gc19e8f2a.el8 2.4 MB/s | 263 kB 00:00 2023-03-24T23:17:22.044 INFO:teuthology.orchestra.run.smithi191.stdout:(35/131): cephadm-18.0.0-3124.gc19e8f2a.el8.noa 1.4 MB/s | 83 kB 00:00 2023-03-24T23:17:22.090 INFO:teuthology.orchestra.run.smithi196.stdout:(84/131): python3-scipy-1.0.0-21.module+el8.5.0 19 MB/s | 14 MB 00:00 2023-03-24T23:17:22.111 INFO:teuthology.orchestra.run.smithi191.stdout:(36/131): cephfs-top-18.0.0-3124.gc19e8f2a.el8. 270 kB/s | 18 kB 00:00 2023-03-24T23:17:22.115 INFO:teuthology.orchestra.run.smithi196.stdout:(85/131): python3-prettytable-0.7.2-14.el8.noar 277 kB/s | 44 kB 00:00 2023-03-24T23:17:22.183 INFO:teuthology.orchestra.run.smithi196.stdout:(86/131): python3-babel-2.5.1-7.el8.noarch.rpm 14 MB/s | 4.8 MB 00:00 2023-03-24T23:17:22.216 INFO:teuthology.orchestra.run.smithi196.stdout:(87/131): python3-werkzeug-0.12.2-4.el8.noarch. 4.5 MB/s | 457 kB 00:00 2023-03-24T23:17:22.242 INFO:teuthology.orchestra.run.smithi196.stdout:(88/131): python3-numpy-f2py-1.14.3-10.el8.x86_ 3.8 MB/s | 225 kB 00:00 2023-03-24T23:17:22.267 INFO:teuthology.orchestra.run.smithi196.stdout:(89/131): uuid-1.6.2-43.el8.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2023-03-24T23:17:22.292 INFO:teuthology.orchestra.run.smithi196.stdout:(90/131): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2023-03-24T23:17:22.318 INFO:teuthology.orchestra.run.smithi196.stdout:(91/131): python3-jsonpointer-1.10-11.el8.noarc 398 kB/s | 20 kB 00:00 2023-03-24T23:17:22.342 INFO:teuthology.orchestra.run.smithi196.stdout:(92/131): python3-pyOpenSSL-19.0.0-1.el8.noarch 2.0 MB/s | 103 kB 00:00 2023-03-24T23:17:22.376 INFO:teuthology.orchestra.run.smithi196.stdout:(93/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-03-24T23:17:22.451 INFO:teuthology.orchestra.run.smithi196.stdout:(94/131): python3-numpy-1.14.3-10.el8.x86_64.rp 10 MB/s | 3.7 MB 00:00 2023-03-24T23:17:22.477 INFO:teuthology.orchestra.run.smithi196.stdout:(95/131): libpq-13.5-1.el8.x86_64.rpm 1.9 MB/s | 198 kB 00:00 2023-03-24T23:17:22.519 INFO:teuthology.orchestra.run.smithi196.stdout:(96/131): jq-1.6-3.el8.x86_64.rpm 4.7 MB/s | 202 kB 00:00 2023-03-24T23:17:22.569 INFO:teuthology.orchestra.run.smithi196.stdout:(97/131): librdkafka-0.11.4-3.el8.x86_64.rpm 6.9 MB/s | 354 kB 00:00 2023-03-24T23:17:22.619 INFO:teuthology.orchestra.run.smithi191.stdout:(37/131): ceph-test-18.0.0-3124.gc19e8f2a.el8.x 7.1 MB/s | 41 MB 00:05 2023-03-24T23:17:22.636 INFO:teuthology.orchestra.run.smithi196.stdout:(98/131): openblas-threads-0.3.15-3.el8.x86_64. 17 MB/s | 4.9 MB 00:00 2023-03-24T23:17:22.687 INFO:teuthology.orchestra.run.smithi191.stdout:(38/131): ceph-mgr-diskprediction-local-18.0.0- 6.2 MB/s | 7.4 MB 00:01 2023-03-24T23:17:22.711 INFO:teuthology.orchestra.run.smithi196.stdout:(99/131): openblas-0.3.15-3.el8.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2023-03-24T23:17:22.712 INFO:teuthology.orchestra.run.smithi191.stdout:(39/131): gperftools-libs-2.7-9.el8.x86_64.rpm 510 kB/s | 306 kB 00:00 2023-03-24T23:17:22.745 INFO:teuthology.orchestra.run.smithi196.stdout:(100/131): postgresql-contrib-10.21-2.module+el 7.3 MB/s | 808 kB 00:00 2023-03-24T23:17:22.787 INFO:teuthology.orchestra.run.smithi196.stdout:(101/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2023-03-24T23:17:22.829 INFO:teuthology.orchestra.run.smithi196.stdout:(102/131): postgresql-10.21-2.module+el8.6.0+15 13 MB/s | 1.5 MB 00:00 2023-03-24T23:17:22.846 INFO:teuthology.orchestra.run.smithi191.stdout:(40/131): liboath-2.6.2-3.el8.x86_64.rpm 235 kB/s | 59 kB 00:00 2023-03-24T23:17:22.854 INFO:teuthology.orchestra.run.smithi196.stdout:(103/131): libselinux-devel-2.9-5.el8.x86_64.rp 2.9 MB/s | 200 kB 00:00 2023-03-24T23:17:22.871 INFO:teuthology.orchestra.run.smithi191.stdout:(41/131): python3-asyncssh-2.7.0-2.el8.noarch.r 2.9 MB/s | 476 kB 00:00 2023-03-24T23:17:22.879 INFO:teuthology.orchestra.run.smithi196.stdout:(104/131): python3-jwt-1.6.1-2.el8.noarch.rpm 848 kB/s | 43 kB 00:00 2023-03-24T23:17:22.905 INFO:teuthology.orchestra.run.smithi196.stdout:(105/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-03-24T23:17:22.913 INFO:teuthology.orchestra.run.smithi191.stdout:(42/131): libunwind-1.3.1-3.el8.x86_64.rpm 332 kB/s | 75 kB 00:00 2023-03-24T23:17:22.921 INFO:teuthology.orchestra.run.smithi196.stdout:(106/131): keyutils-libs-devel-1.5.10-9.el8.x86 1.1 MB/s | 48 kB 00:00 2023-03-24T23:17:22.938 INFO:teuthology.orchestra.run.smithi191.stdout:(43/131): python3-bcrypt-3.1.6-2.el8.1.x86_64.r 473 kB/s | 44 kB 00:00 2023-03-24T23:17:22.946 INFO:teuthology.orchestra.run.smithi196.stdout:(107/131): libsepol-devel-2.9-3.el8.x86_64.rpm 2.0 MB/s | 87 kB 00:00 2023-03-24T23:17:22.988 INFO:teuthology.orchestra.run.smithi191.stdout:(44/131): python3-beautifulsoup4-4.6.3-2.el8.1. 1.5 MB/s | 185 kB 00:00 2023-03-24T23:17:22.989 INFO:teuthology.orchestra.run.smithi196.stdout:(108/131): python3-requests-oauthlib-1.0.0-1.el 638 kB/s | 43 kB 00:00 2023-03-24T23:17:23.014 INFO:teuthology.orchestra.run.smithi191.stdout:(45/131): python3-cachetools-3.1.1-4.el8.noarch 328 kB/s | 33 kB 00:00 2023-03-24T23:17:23.015 INFO:teuthology.orchestra.run.smithi196.stdout:(109/131): python3-pycparser-2.14-14.el8.noarch 1.6 MB/s | 109 kB 00:00 2023-03-24T23:17:23.039 INFO:teuthology.orchestra.run.smithi191.stdout:(46/131): python3-certifi-2018.10.15-7.el8.noar 162 kB/s | 16 kB 00:00 2023-03-24T23:17:23.040 INFO:teuthology.orchestra.run.smithi196.stdout:(110/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.4 MB/s | 229 kB 00:00 2023-03-24T23:17:23.081 INFO:teuthology.orchestra.run.smithi191.stdout:(47/131): python3-cheroot-8.5.2-1.el8.noarch.rp 1.8 MB/s | 173 kB 00:00 2023-03-24T23:17:23.106 INFO:teuthology.orchestra.run.smithi196.stdout:(111/131): postgresql-server-10.21-2.module+el8 9.4 MB/s | 5.1 MB 00:00 2023-03-24T23:17:23.123 INFO:teuthology.orchestra.run.smithi196.stdout:(112/131): libaio-devel-0.3.112-1.el8.x86_64.rp 223 kB/s | 19 kB 00:00 2023-03-24T23:17:23.131 INFO:teuthology.orchestra.run.smithi191.stdout:(48/131): python3-google-auth-1.1.1-10.el8.noar 884 kB/s | 81 kB 00:00 2023-03-24T23:17:23.148 INFO:teuthology.orchestra.run.smithi196.stdout:(113/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 4.1 MB/s | 560 kB 00:00 2023-03-24T23:17:23.173 INFO:teuthology.orchestra.run.smithi196.stdout:(114/131): python3-oauthlib-2.1.0-1.el8.noarch. 2.3 MB/s | 155 kB 00:00 2023-03-24T23:17:23.199 INFO:teuthology.orchestra.run.smithi196.stdout:(115/131): libverto-devel-0.3.0-5.el8.x86_64.rp 356 kB/s | 18 kB 00:00 2023-03-24T23:17:23.206 INFO:teuthology.orchestra.run.smithi191.stdout:(49/131): python3-jaraco-functools-2.0-4.el8.no 234 kB/s | 18 kB 00:00 2023-03-24T23:17:23.232 INFO:teuthology.orchestra.run.smithi196.stdout:(116/131): python3-cryptography-3.2.1-5.el8.x86 5.0 MB/s | 559 kB 00:00 2023-03-24T23:17:23.258 INFO:teuthology.orchestra.run.smithi196.stdout:(117/131): pcre2-devel-10.32-2.el8.x86_64.rpm 7.0 MB/s | 605 kB 00:00 2023-03-24T23:17:23.283 INFO:teuthology.orchestra.run.smithi196.stdout:(118/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 2.8 MB/s | 238 kB 00:00 2023-03-24T23:17:23.308 INFO:teuthology.orchestra.run.smithi196.stdout:(119/131): xfsprogs-devel-5.0.0-10.el8.x86_64.r 1.4 MB/s | 72 kB 00:00 2023-03-24T23:17:23.326 INFO:teuthology.orchestra.run.smithi196.stdout:(120/131): fuse-2.9.7-15.el8.x86_64.rpm 1.9 MB/s | 83 kB 00:00 2023-03-24T23:17:23.351 INFO:teuthology.orchestra.run.smithi196.stdout:(121/131): elfutils-libelf-devel-0.186-1.el8.x8 1.4 MB/s | 60 kB 00:00 2023-03-24T23:17:23.376 INFO:teuthology.orchestra.run.smithi196.stdout:(122/131): libuuid-devel-2.32.1-35.el8.x86_64.r 1.9 MB/s | 98 kB 00:00 2023-03-24T23:17:23.401 INFO:teuthology.orchestra.run.smithi196.stdout:(123/131): zlib-devel-1.2.11-18.el8_5.x86_64.rp 1.1 MB/s | 58 kB 00:00 2023-03-24T23:17:23.426 INFO:teuthology.orchestra.run.smithi196.stdout:(124/131): libcom_err-devel-1.45.6-4.el8.x86_64 767 kB/s | 39 kB 00:00 2023-03-24T23:17:23.448 INFO:teuthology.orchestra.run.smithi191.stdout:(50/131): python3-cherrypy-18.4.0-1.el8.noarch. 885 kB/s | 384 kB 00:00 2023-03-24T23:17:23.460 INFO:teuthology.orchestra.run.smithi196.stdout:(125/131): libgfortran-8.5.0-10.1.el8_6.x86_64. 11 MB/s | 644 kB 00:00 2023-03-24T23:17:23.485 INFO:teuthology.orchestra.run.smithi196.stdout:(126/131): libquadmath-8.5.0-10.1.el8_6.x86_64. 2.9 MB/s | 171 kB 00:00 2023-03-24T23:17:23.499 INFO:teuthology.orchestra.run.smithi191.stdout:(51/131): python3-logutils-0.3.5-11.el8.noarch. 977 kB/s | 49 kB 00:00 2023-03-24T23:17:23.519 INFO:teuthology.orchestra.run.smithi196.stdout:(127/131): NetworkManager-initscripts-updown-1. 2.3 MB/s | 138 kB 00:00 2023-03-24T23:17:23.569 INFO:teuthology.orchestra.run.smithi196.stdout:(128/131): openssl-devel-1.1.1k-7.el8_6.x86_64. 6.9 MB/s | 2.3 MB 00:00 2023-03-24T23:17:23.574 INFO:teuthology.orchestra.run.smithi191.stdout:(52/131): python3-more-itertools-7.2.0-3.el8.no 783 kB/s | 59 kB 00:00 2023-03-24T23:17:23.620 INFO:teuthology.orchestra.run.smithi196.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-03-24T23:17:23.674 INFO:teuthology.orchestra.run.smithi191.stdout:(53/131): python3-natsort-7.1.1-2.el8.noarch.rp 602 kB/s | 60 kB 00:00 2023-03-24T23:17:23.708 INFO:teuthology.orchestra.run.smithi191.stdout:(54/131): python3-kubernetes-11.0.0-6.el8.noarc 2.3 MB/s | 1.2 MB 00:00 2023-03-24T23:17:23.725 INFO:teuthology.orchestra.run.smithi191.stdout:(55/131): python3-jaraco-6.2-6.el8.noarch.rpm 17 kB/s | 11 kB 00:00 2023-03-24T23:17:23.775 INFO:teuthology.orchestra.run.smithi191.stdout:(56/131): python3-repoze-lru-0.7-6.el8.noarch.r 674 kB/s | 34 kB 00:00 2023-03-24T23:17:23.792 INFO:teuthology.orchestra.run.smithi191.stdout:(57/131): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2023-03-24T23:17:23.876 INFO:teuthology.orchestra.run.smithi191.stdout:(58/131): python3-rsa-4.8-1.el8.noarch.rpm 739 kB/s | 61 kB 00:00 2023-03-24T23:17:23.878 INFO:teuthology.orchestra.run.smithi196.stdout:(130/131): librados2-18.0.0-3124.gc19e8f2a.el8. 9.3 MB/s | 3.7 MB 00:00 2023-03-24T23:17:23.909 INFO:teuthology.orchestra.run.smithi191.stdout:(59/131): python3-pecan-1.3.2-9.el8.noarch.rpm 1.2 MB/s | 283 kB 00:00 2023-03-24T23:17:23.926 INFO:teuthology.orchestra.run.smithi191.stdout:(60/131): python3-simplegeneric-0.8.1-17.el8.no 373 kB/s | 19 kB 00:00 2023-03-24T23:17:23.952 INFO:teuthology.orchestra.run.smithi191.stdout:(61/131): python3-routes-2.4.1-12.el8.noarch.rp 1.1 MB/s | 196 kB 00:00 2023-03-24T23:17:23.962 INFO:teuthology.orchestra.run.smithi196.stdout:(131/131): librbd1-18.0.0-3124.gc19e8f2a.el8.x8 8.1 MB/s | 3.6 MB 00:00 2023-03-24T23:17:23.964 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2023-03-24T23:17:23.964 INFO:teuthology.orchestra.run.smithi196.stdout:Total 17 MB/s | 197 MB 00:11 2023-03-24T23:17:23.969 INFO:teuthology.orchestra.run.smithi191.stdout:(62/131): python3-singledispatch-3.4.0.3-18.el8 407 kB/s | 24 kB 00:00 2023-03-24T23:17:23.985 INFO:teuthology.orchestra.run.smithi191.stdout:(63/131): python3-tempora-1.14.1-5.el8.noarch.r 495 kB/s | 29 kB 00:00 2023-03-24T23:17:24.002 INFO:teuthology.orchestra.run.smithi191.stdout:(64/131): python3-trustme-0.6.0-4.el8.noarch.rp 546 kB/s | 27 kB 00:00 2023-03-24T23:17:24.053 INFO:teuthology.orchestra.run.smithi191.stdout:(65/131): python3-websocket-client-0.56.0-5.el8 1.2 MB/s | 61 kB 00:00 2023-03-24T23:17:24.111 INFO:teuthology.orchestra.run.smithi191.stdout:(66/131): python3-webob-1.8.5-1.el8.1.noarch.rp 2.0 MB/s | 251 kB 00:00 2023-03-24T23:17:24.161 INFO:teuthology.orchestra.run.smithi191.stdout:(67/131): python3-webtest-2.0.33-1.el8.noarch.r 791 kB/s | 85 kB 00:00 2023-03-24T23:17:24.187 INFO:teuthology.orchestra.run.smithi191.stdout:(68/131): python3-zc-lockfile-2.0-2.el8.noarch. 304 kB/s | 23 kB 00:00 2023-03-24T23:17:24.212 INFO:teuthology.orchestra.run.smithi191.stdout:(69/131): python3-waitress-1.2.1-2.el8.1.noarch 990 kB/s | 241 kB 00:00 2023-03-24T23:17:24.238 INFO:teuthology.orchestra.run.smithi191.stdout:(70/131): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.4 MB/s | 70 kB 00:00 2023-03-24T23:17:24.280 INFO:teuthology.orchestra.run.smithi191.stdout:(71/131): librabbitmq-0.9.0-3.el8.x86_64.rpm 701 kB/s | 47 kB 00:00 2023-03-24T23:17:24.297 INFO:teuthology.orchestra.run.smithi191.stdout:(72/131): libselinux-devel-2.9-5.el8.x86_64.rpm 3.7 MB/s | 200 kB 00:00 2023-03-24T23:17:24.322 INFO:teuthology.orchestra.run.smithi191.stdout:(73/131): python3-jwt-1.6.1-2.el8.noarch.rpm 1.0 MB/s | 43 kB 00:00 2023-03-24T23:17:24.347 INFO:teuthology.orchestra.run.smithi191.stdout:(74/131): pcre2-utf32-10.32-2.el8.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2023-03-24T23:17:24.372 INFO:teuthology.orchestra.run.smithi191.stdout:(75/131): keyutils-libs-devel-1.5.10-9.el8.x86_ 957 kB/s | 48 kB 00:00 2023-03-24T23:17:24.398 INFO:teuthology.orchestra.run.smithi191.stdout:(76/131): libsepol-devel-2.9-3.el8.x86_64.rpm 1.7 MB/s | 87 kB 00:00 2023-03-24T23:17:24.423 INFO:teuthology.orchestra.run.smithi191.stdout:(77/131): python3-requests-oauthlib-1.0.0-1.el8 848 kB/s | 43 kB 00:00 2023-03-24T23:17:24.448 INFO:teuthology.orchestra.run.smithi191.stdout:(78/131): python3-pycparser-2.14-14.el8.noarch. 2.1 MB/s | 109 kB 00:00 2023-03-24T23:17:24.474 INFO:teuthology.orchestra.run.smithi191.stdout:(79/131): pcre2-utf16-10.32-2.el8.x86_64.rpm 4.5 MB/s | 229 kB 00:00 2023-03-24T23:17:24.507 INFO:teuthology.orchestra.run.smithi191.stdout:(80/131): krb5-devel-1.18.2-14.el8.x86_64.rpm 9.4 MB/s | 560 kB 00:00 2023-03-24T23:17:24.556 INFO:teuthology.orchestra.run.smithi191.stdout:(81/131): libaio-devel-0.3.112-1.el8.x86_64.rpm 229 kB/s | 19 kB 00:00 2023-03-24T23:17:24.583 INFO:teuthology.orchestra.run.smithi191.stdout:(82/131): python3-oauthlib-2.1.0-1.el8.noarch.r 2.0 MB/s | 155 kB 00:00 2023-03-24T23:17:24.617 INFO:teuthology.orchestra.run.smithi191.stdout:(83/131): python3-cryptography-3.2.1-5.el8.x86_ 9.0 MB/s | 559 kB 00:00 2023-03-24T23:17:24.642 INFO:teuthology.orchestra.run.smithi191.stdout:(84/131): libverto-devel-0.3.0-5.el8.x86_64.rpm 305 kB/s | 18 kB 00:00 2023-03-24T23:17:24.676 INFO:teuthology.orchestra.run.smithi191.stdout:(85/131): pcre2-devel-10.32-2.el8.x86_64.rpm 10 MB/s | 605 kB 00:00 2023-03-24T23:17:24.709 INFO:teuthology.orchestra.run.smithi191.stdout:(86/131): python3-cffi-1.11.5-5.el8.x86_64.rpm 3.5 MB/s | 238 kB 00:00 2023-03-24T23:17:24.751 INFO:teuthology.orchestra.run.smithi191.stdout:(87/131): xfsprogs-devel-5.0.0-10.el8.x86_64.rp 1.7 MB/s | 72 kB 00:00 2023-03-24T23:17:24.785 INFO:teuthology.orchestra.run.smithi191.stdout:(88/131): thrift-0.13.0-2.el8.x86_64.rpm 2.8 MB/s | 1.7 MB 00:00 2023-03-24T23:17:24.800 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2023-03-24T23:17:24.802 INFO:teuthology.orchestra.run.smithi191.stdout:(89/131): fuse-2.9.7-15.el8.x86_64.rpm 1.6 MB/s | 83 kB 00:00 2023-03-24T23:17:24.869 INFO:teuthology.orchestra.run.smithi191.stdout:(90/131): openssl-devel-1.1.1k-7.el8_6.x86_64.r 12 MB/s | 2.3 MB 00:00 2023-03-24T23:17:24.894 INFO:teuthology.orchestra.run.smithi191.stdout:(91/131): elfutils-libelf-devel-0.186-1.el8.x86 551 kB/s | 60 kB 00:00 2023-03-24T23:17:24.919 INFO:teuthology.orchestra.run.smithi191.stdout:(92/131): libuuid-devel-2.32.1-35.el8.x86_64.rp 879 kB/s | 98 kB 00:00 2023-03-24T23:17:24.920 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2023-03-24T23:17:24.921 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2023-03-24T23:17:24.970 INFO:teuthology.orchestra.run.smithi191.stdout:(93/131): zlib-devel-1.2.11-18.el8_5.x86_64.rpm 573 kB/s | 58 kB 00:00 2023-03-24T23:17:24.995 INFO:teuthology.orchestra.run.smithi191.stdout:(94/131): libcom_err-devel-1.45.6-4.el8.x86_64. 383 kB/s | 39 kB 00:00 2023-03-24T23:17:25.020 INFO:teuthology.orchestra.run.smithi191.stdout:(95/131): libquadmath-8.5.0-10.1.el8_6.x86_64.r 3.3 MB/s | 171 kB 00:00 2023-03-24T23:17:25.046 INFO:teuthology.orchestra.run.smithi191.stdout:(96/131): NetworkManager-initscripts-updown-1.3 2.7 MB/s | 138 kB 00:00 2023-03-24T23:17:25.079 INFO:teuthology.orchestra.run.smithi191.stdout:(97/131): libgfortran-8.5.0-10.1.el8_6.x86_64.r 3.9 MB/s | 644 kB 00:00 2023-03-24T23:17:25.104 INFO:teuthology.orchestra.run.smithi191.stdout:(98/131): python3-pyasn1-modules-0.3.7-6.el8.no 1.8 MB/s | 110 kB 00:00 2023-03-24T23:17:25.122 INFO:teuthology.orchestra.run.smithi191.stdout:(99/131): python3-pytz-2017.2-9.el8.noarch.rpm 1.3 MB/s | 54 kB 00:00 2023-03-24T23:17:25.147 INFO:teuthology.orchestra.run.smithi191.stdout:(100/131): lttng-ust-2.8.1-11.el8.x86_64.rpm 6.0 MB/s | 259 kB 00:00 2023-03-24T23:17:25.180 INFO:teuthology.orchestra.run.smithi191.stdout:(101/131): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-03-24T23:17:25.197 INFO:teuthology.orchestra.run.smithi191.stdout:(102/131): python3-markupsafe-0.23-19.el8.x86_6 776 kB/s | 39 kB 00:00 2023-03-24T23:17:25.231 INFO:teuthology.orchestra.run.smithi191.stdout:(103/131): bison-3.0.4-10.el8.x86_64.rpm 13 MB/s | 688 kB 00:00 2023-03-24T23:17:25.264 INFO:teuthology.orchestra.run.smithi191.stdout:(104/131): python3-jinja2-2.10.1-3.el8.noarch.r 7.9 MB/s | 538 kB 00:00 2023-03-24T23:17:25.290 INFO:teuthology.orchestra.run.smithi191.stdout:(105/131): python3-mako-1.0.6-13.el8.noarch.rpm 2.6 MB/s | 157 kB 00:00 2023-03-24T23:17:25.315 INFO:teuthology.orchestra.run.smithi191.stdout:(106/131): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-03-24T23:17:25.374 INFO:teuthology.orchestra.run.smithi191.stdout:(107/131): automake-1.16.1-7.el8.noarch.rpm 8.3 MB/s | 713 kB 00:00 2023-03-24T23:17:25.416 INFO:teuthology.orchestra.run.smithi191.stdout:(108/131): python3-jsonpatch-1.21-2.el8.noarch. 638 kB/s | 27 kB 00:00 2023-03-24T23:17:25.458 INFO:teuthology.orchestra.run.smithi191.stdout:(109/131): flex-2.6.1-9.el8.x86_64.rpm 7.5 MB/s | 320 kB 00:00 2023-03-24T23:17:25.500 INFO:teuthology.orchestra.run.smithi191.stdout:(110/131): python3-prettytable-0.7.2-14.el8.noa 1.0 MB/s | 44 kB 00:00 2023-03-24T23:17:25.583 INFO:teuthology.orchestra.run.smithi191.stdout:(111/131): python3-babel-2.5.1-7.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2023-03-24T23:17:25.625 INFO:teuthology.orchestra.run.smithi191.stdout:(112/131): python3-werkzeug-0.12.2-4.el8.noarch 11 MB/s | 457 kB 00:00 2023-03-24T23:17:25.668 INFO:teuthology.orchestra.run.smithi191.stdout:(113/131): python3-numpy-f2py-1.14.3-10.el8.x86 5.3 MB/s | 225 kB 00:00 2023-03-24T23:17:25.756 INFO:teuthology.orchestra.run.smithi191.stdout:(114/131): python3-numpy-1.14.3-10.el8.x86_64.r 15 MB/s | 3.7 MB 00:00 2023-03-24T23:17:25.784 INFO:teuthology.orchestra.run.smithi191.stdout:(115/131): uuid-1.6.2-43.el8.x86_64.rpm 548 kB/s | 64 kB 00:00 2023-03-24T23:17:25.809 INFO:teuthology.orchestra.run.smithi191.stdout:(116/131): python3-pyasn1-0.3.7-6.el8.noarch.rp 2.3 MB/s | 126 kB 00:00 2023-03-24T23:17:25.835 INFO:teuthology.orchestra.run.smithi191.stdout:(117/131): python3-jsonpointer-1.10-11.el8.noar 399 kB/s | 20 kB 00:00 2023-03-24T23:17:25.860 INFO:teuthology.orchestra.run.smithi191.stdout:(118/131): python3-pyOpenSSL-19.0.0-1.el8.noarc 2.0 MB/s | 103 kB 00:00 2023-03-24T23:17:25.894 INFO:teuthology.orchestra.run.smithi191.stdout:(119/131): libtool-2.4.6-25.el8.x86_64.rpm 12 MB/s | 709 kB 00:00 2023-03-24T23:17:25.995 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2023-03-24T23:17:25.998 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2023-03-24T23:17:26.269 INFO:teuthology.orchestra.run.smithi191.stdout:(120/131): python3-scipy-1.0.0-21.module+el8.5. 11 MB/s | 14 MB 00:01 2023-03-24T23:17:26.294 INFO:teuthology.orchestra.run.smithi191.stdout:(121/131): libpq-13.5-1.el8.x86_64.rpm 494 kB/s | 198 kB 00:00 2023-03-24T23:17:26.336 INFO:teuthology.orchestra.run.smithi191.stdout:(122/131): jq-1.6-3.el8.x86_64.rpm 4.8 MB/s | 202 kB 00:00 2023-03-24T23:17:26.378 INFO:teuthology.orchestra.run.smithi191.stdout:(123/131): librdkafka-0.11.4-3.el8.x86_64.rpm 8.3 MB/s | 354 kB 00:00 2023-03-24T23:17:26.462 INFO:teuthology.orchestra.run.smithi191.stdout:(124/131): openblas-threads-0.3.15-3.el8.x86_64 8.1 MB/s | 4.9 MB 00:00 2023-03-24T23:17:26.579 INFO:teuthology.orchestra.run.smithi191.stdout:(125/131): openblas-0.3.15-3.el8.x86_64.rpm 15 MB/s | 4.7 MB 00:00 2023-03-24T23:17:26.613 INFO:teuthology.orchestra.run.smithi191.stdout:(126/131): postgresql-contrib-10.21-2.module+el 5.2 MB/s | 808 kB 00:00 2023-03-24T23:17:26.663 INFO:teuthology.orchestra.run.smithi191.stdout:(127/131): postgresql-10.21-2.module+el8.6.0+15 18 MB/s | 1.5 MB 00:00 2023-03-24T23:17:26.747 INFO:teuthology.orchestra.run.smithi191.stdout:(128/131): postgresql-server-10.21-2.module+el8 14 MB/s | 5.1 MB 00:00 2023-03-24T23:17:26.797 INFO:teuthology.orchestra.run.smithi191.stdout:(129/131): iproute-5.15.0-4.el8_6.1.x86_64.rpm 16 MB/s | 798 kB 00:00 2023-03-24T23:17:27.047 INFO:teuthology.orchestra.run.smithi191.stdout:(130/131): librados2-18.0.0-3124.gc19e8f2a.el8. 8.4 MB/s | 3.7 MB 00:00 2023-03-24T23:17:27.189 INFO:teuthology.orchestra.run.smithi191.stdout:(131/131): librbd1-18.0.0-3124.gc19e8f2a.el8.x8 6.8 MB/s | 3.6 MB 00:00 2023-03-24T23:17:27.192 INFO:teuthology.orchestra.run.smithi191.stdout:-------------------------------------------------------------------------------- 2023-03-24T23:17:27.192 INFO:teuthology.orchestra.run.smithi191.stdout:Total 14 MB/s | 197 MB 00:14 2023-03-24T23:17:28.006 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction check 2023-03-24T23:17:28.126 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction check succeeded. 2023-03-24T23:17:28.126 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction test 2023-03-24T23:17:29.203 INFO:teuthology.orchestra.run.smithi191.stdout:Transaction test succeeded. 2023-03-24T23:17:29.207 INFO:teuthology.orchestra.run.smithi191.stdout:Running transaction 2023-03-24T23:17:30.806 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2023-03-24T23:17:30.993 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-24T23:17:31.192 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-24T23:17:31.393 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 3/134 2023-03-24T23:17:31.570 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-24T23:17:31.674 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 4/134 2023-03-24T23:17:31.887 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-24T23:17:32.289 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 5/134 2023-03-24T23:17:32.478 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-24T23:17:32.878 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-24T23:17:33.053 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-24T23:17:33.161 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-24T23:17:33.361 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-24T23:17:33.442 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-24T23:17:33.636 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 9/134 2023-03-24T23:17:33.856 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-24T23:17:34.047 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-24T23:17:34.422 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-24T23:17:34.519 INFO:teuthology.orchestra.run.smithi191.stdout: Preparing : 1/1 2023-03-24T23:17:34.689 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : thrift-0.13.0-2.el8.x86_64 1/134 2023-03-24T23:17:34.690 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-24T23:17:34.861 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 14/134 2023-03-24T23:17:34.906 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/134 2023-03-24T23:17:34.955 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 14/134 2023-03-24T23:17:35.066 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-24T23:17:35.136 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 15/134 2023-03-24T23:17:35.196 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 3/134 2023-03-24T23:17:35.205 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 15/134 2023-03-24T23:17:35.360 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-24T23:17:35.416 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-24T23:17:35.505 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 16/134 2023-03-24T23:17:35.669 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-24T23:17:35.817 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 4/134 2023-03-24T23:17:36.107 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 5/134 2023-03-24T23:17:36.285 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 17/134 2023-03-24T23:17:36.444 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-24T23:17:36.488 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 5/134 2023-03-24T23:17:36.683 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-24T23:17:36.798 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-24T23:17:37.024 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-24T23:17:37.034 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 18/134 2023-03-24T23:17:37.119 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-24T23:17:37.210 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.15-3.el8.x86_64 19/134 2023-03-24T23:17:37.897 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-24T23:17:37.904 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 19/134 2023-03-24T23:17:38.101 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librabbitmq-0.9.0-3.el8.x86_64 9/134 2023-03-24T23:17:38.185 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 20/134 2023-03-24T23:17:38.393 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libpq-13.5-1.el8.x86_64 10/134 2023-03-24T23:17:38.411 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 21/134 2023-03-24T23:17:38.614 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 22/134 2023-03-24T23:17:38.637 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 11/134 2023-03-24T23:17:38.783 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 23/134 2023-03-24T23:17:38.983 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/134 2023-03-24T23:17:39.061 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 12/134 2023-03-24T23:17:39.195 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-24T23:17:39.302 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-10.21-2.module+el8.6.0+15342+53518fac 13/134 2023-03-24T23:17:39.350 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8. 26/134 2023-03-24T23:17:39.496 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el 27/134 2023-03-24T23:17:39.499 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 14/134 2023-03-24T23:17:39.601 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 14/134 2023-03-24T23:17:39.806 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 15/134 2023-03-24T23:17:39.952 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 15/134 2023-03-24T23:17:40.186 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 16/134 2023-03-24T23:17:40.371 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 17/134 2023-03-24T23:17:40.385 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_6 28/134 2023-03-24T23:17:40.595 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 18/134 2023-03-24T23:17:40.665 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 29/134 2023-03-24T23:17:40.748 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-24T23:17:40.884 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libquadmath-8.5.0-10.1.el8_6.x86_64 19/134 2023-03-24T23:17:40.919 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 30/134 2023-03-24T23:17:41.048 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-24T23:17:41.165 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 31/134 2023-03-24T23:17:41.559 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-24T23:17:41.667 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libgfortran-8.5.0-10.1.el8_6.x86_64 20/134 2023-03-24T23:17:41.874 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-24T23:17:42.458 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-threads-0.3.15-3.el8.x86_64 21/134 2023-03-24T23:17:42.665 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openblas-0.3.15-3.el8.x86_64 22/134 2023-03-24T23:17:43.309 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 32/134 2023-03-24T23:17:43.348 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: openblas-0.3.15-3.el8.x86_64 22/134 2023-03-24T23:17:43.619 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 33/134 2023-03-24T23:17:43.715 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 23/134 2023-03-24T23:17:43.846 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-24T23:17:43.929 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : zlib-devel-1.2.11-18.el8_5.x86_64 24/134 2023-03-24T23:17:44.036 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-24T23:17:44.189 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 36/134 2023-03-24T23:17:44.191 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 25/134 2023-03-24T23:17:44.338 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_6 37/134 2023-03-24T23:17:44.362 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8. 26/134 2023-03-24T23:17:44.503 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 38/134 2023-03-24T23:17:44.522 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el 27/134 2023-03-24T23:17:44.670 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 39/134 2023-03-24T23:17:44.769 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_6 28/134 2023-03-24T23:17:44.845 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.7-15.el8.x86_64 40/134 2023-03-24T23:17:45.006 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-24T23:17:45.020 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 41/134 2023-03-24T23:17:45.206 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 42/134 2023-03-24T23:17:45.405 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 43/134 2023-03-24T23:17:45.718 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 44/134 2023-03-24T23:17:46.020 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 45/134 2023-03-24T23:17:46.252 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 46/134 2023-03-24T23:17:46.471 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 47/134 2023-03-24T23:17:46.705 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 48/134 2023-03-24T23:17:46.780 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 29/134 2023-03-24T23:17:47.237 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 49/134 2023-03-24T23:17:47.466 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 50/134 2023-03-24T23:17:47.643 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 51/134 2023-03-24T23:17:47.816 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 30/134 2023-03-24T23:17:47.838 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 52/134 2023-03-24T23:17:47.989 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 53/134 2023-03-24T23:17:48.153 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 54/134 2023-03-24T23:17:48.154 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 31/134 2023-03-24T23:17:48.407 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 32/134 2023-03-24T23:17:48.447 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 55/134 2023-03-24T23:17:48.650 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 33/134 2023-03-24T23:17:48.686 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 56/134 2023-03-24T23:17:48.858 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rsa-4.8-1.el8.noarch 34/134 2023-03-24T23:17:48.921 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 57/134 2023-03-24T23:17:49.066 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 35/134 2023-03-24T23:17:49.148 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 58/134 2023-03-24T23:17:49.244 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 36/134 2023-03-24T23:17:49.323 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 59/134 2023-03-24T23:17:49.420 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_6 37/134 2023-03-24T23:17:49.484 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 60/134 2023-03-24T23:17:49.587 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 38/134 2023-03-24T23:17:49.675 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : uuid-1.6.2-43.el8.x86_64 61/134 2023-03-24T23:17:49.742 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 61/134 2023-03-24T23:17:49.798 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 39/134 2023-03-24T23:17:49.961 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 62/134 2023-03-24T23:17:49.996 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : uuid-1.6.2-43.el8.x86_64 40/134 2023-03-24T23:17:50.096 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: uuid-1.6.2-43.el8.x86_64 40/134 2023-03-24T23:17:50.156 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : automake-1.16.1-7.el8.noarch 63/134 2023-03-24T23:17:50.316 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-24T23:17:50.317 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 41/134 2023-03-24T23:17:50.395 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 64/134 2023-03-24T23:17:50.550 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : automake-1.16.1-7.el8.noarch 42/134 2023-03-24T23:17:50.603 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-3.el8.x86_64 65/134 2023-03-24T23:17:50.705 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-24T23:17:50.766 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 43/134 2023-03-24T23:17:50.792 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 66/134 2023-03-24T23:17:50.932 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-24T23:17:51.009 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : jq-1.6-3.el8.x86_64 44/134 2023-03-24T23:17:51.102 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-24T23:17:51.175 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 45/134 2023-03-24T23:17:51.346 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-24T23:17:51.365 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcom_err-devel-1.45.6-4.el8.x86_64 46/134 2023-03-24T23:17:51.538 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libuuid-devel-2.32.1-35.el8.x86_64 47/134 2023-03-24T23:17:51.556 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-24T23:17:51.712 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-24T23:17:51.749 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : fuse-2.9.7-15.el8.x86_64 48/134 2023-03-24T23:17:51.884 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-24T23:17:51.892 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libverto-devel-0.3.0-5.el8.x86_64 49/134 2023-03-24T23:17:52.065 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-24T23:17:52.112 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-utf16-10.32-2.el8.x86_64 50/134 2023-03-24T23:17:52.260 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-24T23:17:52.302 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 51/134 2023-03-24T23:17:52.426 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-24T23:17:52.569 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-24T23:17:52.673 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cffi-1.11.5-5.el8.x86_64 52/134 2023-03-24T23:17:52.705 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-24T23:17:52.910 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-24T23:17:52.990 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cryptography-3.2.1-5.el8.x86_64 53/134 2023-03-24T23:17:53.153 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-24T23:17:53.207 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 54/134 2023-03-24T23:17:53.322 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-24T23:17:53.482 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 55/134 2023-03-24T23:17:53.563 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-24T23:17:53.768 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 56/134 2023-03-24T23:17:53.798 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-24T23:17:53.965 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-24T23:17:54.095 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 57/134 2023-03-24T23:17:54.166 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-24T23:17:54.320 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 58/134 2023-03-24T23:17:54.551 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 59/134 2023-03-24T23:17:54.785 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 60/134 2023-03-24T23:17:55.001 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libsepol-devel-2.9-3.el8.x86_64 61/134 2023-03-24T23:17:55.166 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : keyutils-libs-devel-1.5.10-9.el8.x86_64 62/134 2023-03-24T23:17:55.273 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-24T23:17:55.492 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-utf32-10.32-2.el8.x86_64 63/134 2023-03-24T23:17:55.527 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-24T23:17:55.748 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-24T23:17:55.775 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : pcre2-devel-10.32-2.el8.x86_64 64/134 2023-03-24T23:17:56.015 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-24T23:17:56.042 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libselinux-devel-2.9-5.el8.x86_64 65/134 2023-03-24T23:17:56.263 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : krb5-devel-1.18.2-14.el8.x86_64 66/134 2023-03-24T23:17:56.302 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-24T23:17:56.445 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 67/134 2023-03-24T23:17:56.478 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-24T23:17:56.632 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 68/134 2023-03-24T23:17:56.934 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 69/134 2023-03-24T23:17:57.169 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 70/134 2023-03-24T23:17:57.287 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-24T23:17:57.349 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 71/134 2023-03-24T23:17:57.448 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 92/134 2023-03-24T23:17:57.514 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 72/134 2023-03-24T23:17:57.521 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 92/134 2023-03-24T23:17:57.737 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 73/134 2023-03-24T23:17:57.782 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 93/134 2023-03-24T23:17:57.949 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 74/134 2023-03-24T23:17:58.166 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 75/134 2023-03-24T23:17:58.323 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 76/134 2023-03-24T23:17:58.485 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 77/134 2023-03-24T23:17:58.718 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 78/134 2023-03-24T23:17:58.982 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 79/134 2023-03-24T23:17:59.176 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 80/134 2023-03-24T23:17:59.467 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-portend-2.6-1.el8.noarch 81/134 2023-03-24T23:17:59.728 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 82/134 2023-03-24T23:17:59.928 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 83/134 2023-03-24T23:18:00.185 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 84/134 2023-03-24T23:18:01.232 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 85/134 2023-03-24T23:18:01.330 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-24T23:18:01.515 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 86/134 2023-03-24T23:18:01.556 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-24T23:18:01.795 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 87/134 2023-03-24T23:18:02.094 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 88/134 2023-03-24T23:18:02.138 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-24T23:18:02.393 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 89/134 2023-03-24T23:18:02.424 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 95/134 2023-03-24T23:18:02.492 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 95/134 2023-03-24T23:18:02.603 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 90/134 2023-03-24T23:18:02.614 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 96/134 2023-03-24T23:18:03.363 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 91/134 2023-03-24T23:18:03.536 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 92/134 2023-03-24T23:18:03.599 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 92/134 2023-03-24T23:18:03.795 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 93/134 2023-03-24T23:18:07.335 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-24T23:18:07.585 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-24T23:18:08.170 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 94/134 2023-03-24T23:18:08.480 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 95/134 2023-03-24T23:18:08.561 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 95/134 2023-03-24T23:18:08.760 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 96/134 2023-03-24T23:18:21.398 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 96/134 2023-03-24T23:18:21.399 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2023-03-24T23:18:21.399 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2023-03-24T23:18:21.399 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /dev 2023-03-24T23:18:21.399 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /run 2023-03-24T23:18:21.400 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2023-03-24T23:18:21.400 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2023-03-24T23:18:21.400 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2023-03-24T23:18:21.400 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2023-03-24T23:18:21.401 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /dev 2023-03-24T23:18:21.401 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:18:21.760 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 97/134 2023-03-24T23:18:23.316 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 97/134 2023-03-24T23:18:23.487 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 98/134 2023-03-24T23:18:23.709 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 98/134 2023-03-24T23:18:23.852 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 99/134 2023-03-24T23:18:24.300 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 99/134 2023-03-24T23:18:24.485 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 100/134 2023-03-24T23:18:24.503 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 100/134 2023-03-24T23:18:26.123 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 101/134 2023-03-24T23:18:26.322 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 101/134 2023-03-24T23:18:26.432 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.e 102/134 2023-03-24T23:18:26.640 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a. 103/134 2023-03-24T23:18:26.817 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 104/134 2023-03-24T23:18:27.029 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 104/134 2023-03-24T23:18:27.319 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 105/134 2023-03-24T23:18:27.720 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 96/134 2023-03-24T23:18:27.720 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /sys 2023-03-24T23:18:27.720 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /proc 2023-03-24T23:18:27.720 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2023-03-24T23:18:27.720 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /run 2023-03-24T23:18:27.720 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /mnt 2023-03-24T23:18:27.721 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /var/tmp 2023-03-24T23:18:27.721 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /home 2023-03-24T23:18:27.721 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /tmp 2023-03-24T23:18:27.721 INFO:teuthology.orchestra.run.smithi191.stdout:skipping the directory /dev 2023-03-24T23:18:27.721 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:18:28.099 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 97/134 2023-03-24T23:18:28.275 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 105/134 2023-03-24T23:18:28.417 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 106/134 2023-03-24T23:18:28.507 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 106/134 2023-03-24T23:18:28.855 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el 107/134 2023-03-24T23:18:29.028 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 108/134 2023-03-24T23:18:29.063 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 108/134 2023-03-24T23:18:29.234 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 109/134 2023-03-24T23:18:29.246 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 109/134 2023-03-24T23:18:29.655 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 97/134 2023-03-24T23:18:29.884 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 98/134 2023-03-24T23:18:30.108 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 98/134 2023-03-24T23:18:30.281 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 99/134 2023-03-24T23:18:30.472 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 110/134 2023-03-24T23:18:30.674 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 111/134 2023-03-24T23:18:30.715 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 111/134 2023-03-24T23:18:30.730 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 99/134 2023-03-24T23:18:30.859 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 112/134 2023-03-24T23:18:30.900 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 112/134 2023-03-24T23:18:30.931 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 100/134 2023-03-24T23:18:30.948 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 100/134 2023-03-24T23:18:31.042 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 113/134 2023-03-24T23:18:31.386 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 113/134 2023-03-24T23:18:31.525 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 114/134 2023-03-24T23:18:32.592 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 101/134 2023-03-24T23:18:32.811 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 101/134 2023-03-24T23:18:32.937 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.e 102/134 2023-03-24T23:18:33.170 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a. 103/134 2023-03-24T23:18:33.405 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 104/134 2023-03-24T23:18:33.617 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 104/134 2023-03-24T23:18:33.883 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 105/134 2023-03-24T23:18:34.841 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 105/134 2023-03-24T23:18:35.038 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 106/134 2023-03-24T23:18:35.129 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 106/134 2023-03-24T23:18:35.430 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 114/134 2023-03-24T23:18:35.494 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el 107/134 2023-03-24T23:18:35.683 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 108/134 2023-03-24T23:18:35.718 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 108/134 2023-03-24T23:18:35.724 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 115/134 2023-03-24T23:18:35.891 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libtool-2.4.6-25.el8.x86_64 116/134 2023-03-24T23:18:35.938 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 109/134 2023-03-24T23:18:35.950 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 109/134 2023-03-24T23:18:36.085 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 116/134 2023-03-24T23:18:36.965 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 117/134 2023-03-24T23:18:37.194 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 110/134 2023-03-24T23:18:37.263 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 118/134 2023-03-24T23:18:37.420 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 119/134 2023-03-24T23:18:37.433 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 111/134 2023-03-24T23:18:37.472 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 111/134 2023-03-24T23:18:37.603 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-24T23:18:37.630 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 112/134 2023-03-24T23:18:37.672 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 112/134 2023-03-24T23:18:37.761 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_ 121/134 2023-03-24T23:18:37.855 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 113/134 2023-03-24T23:18:37.875 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-24T23:18:38.199 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 113/134 2023-03-24T23:18:38.346 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 114/134 2023-03-24T23:18:39.964 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-24T23:18:40.207 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-24T23:18:40.224 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-24T23:18:40.425 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 124/134 2023-03-24T23:18:40.673 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 125/134 2023-03-24T23:18:40.955 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 126/134 2023-03-24T23:18:41.258 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-24T23:18:41.407 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-24T23:18:41.415 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-24T23:18:41.632 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 129/134 2023-03-24T23:18:41.798 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flex-2.6.1-9.el8.x86_64 130/134 2023-03-24T23:18:41.878 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 130/134 2023-03-24T23:18:42.073 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : bison-3.0.4-10.el8.x86_64 131/134 2023-03-24T23:18:42.085 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 131/134 2023-03-24T23:18:42.086 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-24T23:18:42.245 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 114/134 2023-03-24T23:18:42.259 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-24T23:18:42.260 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-24T23:18:42.470 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-24T23:18:42.674 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-24T23:18:42.958 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-24T23:18:42.959 INFO:teuthology.orchestra.run.smithi196.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-24T23:18:42.959 INFO:teuthology.orchestra.run.smithi196.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-24T23:18:42.959 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:18:43.157 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 115/134 2023-03-24T23:18:43.176 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 1/134 2023-03-24T23:18:43.176 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2/134 2023-03-24T23:18:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 3/134 2023-03-24T23:18:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 4/134 2023-03-24T23:18:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 5/134 2023-03-24T23:18:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-24T23:18:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-24T23:18:43.177 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-24T23:18:43.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 9/134 2023-03-24T23:18:43.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 10/134 2023-03-24T23:18:43.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 11/134 2023-03-24T23:18:43.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 12/134 2023-03-24T23:18:43.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 13/134 2023-03-24T23:18:43.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_ 14/134 2023-03-24T23:18:43.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 15/134 2023-03-24T23:18:43.178 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 16/134 2023-03-24T23:18:43.179 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_6 17/134 2023-03-24T23:18:43.179 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 18/134 2023-03-24T23:18:43.179 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 19/134 2023-03-24T23:18:43.179 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el 20/134 2023-03-24T23:18:43.179 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8. 21/134 2023-03-24T23:18:43.179 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_6 22/134 2023-03-24T23:18:43.179 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 23/134 2023-03-24T23:18:43.179 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 24/134 2023-03-24T23:18:43.180 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 25/134 2023-03-24T23:18:43.180 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 26/134 2023-03-24T23:18:43.186 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 27/134 2023-03-24T23:18:43.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 28/134 2023-03-24T23:18:43.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a. 29/134 2023-03-24T23:18:43.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 30/134 2023-03-24T23:18:43.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 31/134 2023-03-24T23:18:43.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 32/134 2023-03-24T23:18:43.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el 33/134 2023-03-24T23:18:43.187 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 34/134 2023-03-24T23:18:43.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.e 35/134 2023-03-24T23:18:43.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 36/134 2023-03-24T23:18:43.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 37/134 2023-03-24T23:18:43.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 38/134 2023-03-24T23:18:43.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-24T23:18:43.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-24T23:18:43.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-24T23:18:43.188 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-24T23:18:43.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-24T23:18:43.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-24T23:18:43.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-24T23:18:43.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-24T23:18:43.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-24T23:18:43.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-24T23:18:43.189 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-24T23:18:43.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-24T23:18:43.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-24T23:18:43.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-24T23:18:43.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-24T23:18:43.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-24T23:18:43.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-24T23:18:43.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-24T23:18:43.190 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-24T23:18:43.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-24T23:18:43.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-24T23:18:43.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-24T23:18:43.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-24T23:18:43.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-24T23:18:43.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-24T23:18:43.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-24T23:18:43.191 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-24T23:18:43.192 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-24T23:18:43.192 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-24T23:18:43.192 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-24T23:18:43.192 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-24T23:18:43.192 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-24T23:18:43.192 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-24T23:18:43.192 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 72/134 2023-03-24T23:18:43.193 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 73/134 2023-03-24T23:18:43.193 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 74/134 2023-03-24T23:18:43.193 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 75/134 2023-03-24T23:18:43.193 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 76/134 2023-03-24T23:18:43.193 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 77/134 2023-03-24T23:18:43.193 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : bison-3.0.4-10.el8.x86_64 78/134 2023-03-24T23:18:43.193 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 79/134 2023-03-24T23:18:43.193 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 80/134 2023-03-24T23:18:43.194 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 81/134 2023-03-24T23:18:43.194 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : automake-1.16.1-7.el8.noarch 82/134 2023-03-24T23:18:43.194 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 83/134 2023-03-24T23:18:43.194 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 84/134 2023-03-24T23:18:43.194 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flex-2.6.1-9.el8.x86_64 85/134 2023-03-24T23:18:43.194 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 86/134 2023-03-24T23:18:43.194 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 87/134 2023-03-24T23:18:43.194 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 88/134 2023-03-24T23:18:43.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 89/134 2023-03-24T23:18:43.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 90/134 2023-03-24T23:18:43.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 91/134 2023-03-24T23:18:43.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 92/134 2023-03-24T23:18:43.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 93/134 2023-03-24T23:18:43.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 94/134 2023-03-24T23:18:43.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 95/134 2023-03-24T23:18:43.195 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpq-13.5-1.el8.x86_64 96/134 2023-03-24T23:18:43.196 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 97/134 2023-03-24T23:18:43.196 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-3.el8.x86_64 98/134 2023-03-24T23:18:43.196 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 99/134 2023-03-24T23:18:43.196 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 100/134 2023-03-24T23:18:43.196 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 101/134 2023-03-24T23:18:43.196 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 102/134 2023-03-24T23:18:43.196 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 103/134 2023-03-24T23:18:43.197 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 104/134 2023-03-24T23:18:43.197 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 105/134 2023-03-24T23:18:43.197 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 106/134 2023-03-24T23:18:43.197 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 107/134 2023-03-24T23:18:43.197 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 108/134 2023-03-24T23:18:43.197 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 109/134 2023-03-24T23:18:43.197 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 110/134 2023-03-24T23:18:43.197 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 111/134 2023-03-24T23:18:43.198 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 112/134 2023-03-24T23:18:43.198 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 113/134 2023-03-24T23:18:43.198 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 114/134 2023-03-24T23:18:43.198 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 115/134 2023-03-24T23:18:43.198 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 116/134 2023-03-24T23:18:43.198 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 117/134 2023-03-24T23:18:43.198 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 118/134 2023-03-24T23:18:43.198 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 119/134 2023-03-24T23:18:43.199 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 120/134 2023-03-24T23:18:43.199 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 121/134 2023-03-24T23:18:43.199 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-24T23:18:43.199 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 123/134 2023-03-24T23:18:43.199 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 124/134 2023-03-24T23:18:43.199 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 125/134 2023-03-24T23:18:43.199 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 126/134 2023-03-24T23:18:43.200 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 127/134 2023-03-24T23:18:43.200 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 128/134 2023-03-24T23:18:43.200 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 129/134 2023-03-24T23:18:43.200 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-24T23:18:43.200 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 131/134 2023-03-24T23:18:43.200 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-24T23:18:43.200 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-24T23:18:43.518 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : openssl-devel-1:1.1.1k-7.el8_6.x86_64 116/134 2023-03-24T23:18:43.716 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 117/134 2023-03-24T23:18:43.997 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : xfsprogs-devel-5.0.0-10.el8.x86_64 118/134 2023-03-24T23:18:44.195 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libtool-2.4.6-25.el8.x86_64 119/134 2023-03-24T23:18:44.390 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: libtool-2.4.6-25.el8.x86_64 119/134 2023-03-24T23:18:44.649 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 120/134 2023-03-24T23:18:44.816 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_ 121/134 2023-03-24T23:18:44.988 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : elfutils-libelf-devel-0.186-1.el8.x86_64 122/134 2023-03-24T23:18:45.849 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-24T23:18:45.849 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:44 ago on Fri 24 Mar 2023 11:17:01 PM UTC. 2023-03-24T23:18:47.161 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-24T23:18:47.445 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-24T23:18:47.462 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: postgresql-server-10.21-2.module+el8.6.0+15342+5 123/134 2023-03-24T23:18:47.678 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 124/134 2023-03-24T23:18:47.928 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 125/134 2023-03-24T23:18:48.199 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 126/134 2023-03-24T23:18:48.597 INFO:teuthology.orchestra.run.smithi191.stdout: Upgrading : iproute-5.15.0-4.el8_6.1.x86_64 127/134 2023-03-24T23:18:48.675 INFO:teuthology.orchestra.run.smithi196.stdout:Installed products updated. 2023-03-24T23:18:48.761 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : flex-2.6.1-9.el8.x86_64 128/134 2023-03-24T23:18:48.840 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: flex-2.6.1-9.el8.x86_64 128/134 2023-03-24T23:18:48.920 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:18:48.920 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2023-03-24T23:18:48.920 INFO:teuthology.orchestra.run.smithi196.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-24T23:18:48.920 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.921 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.921 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2023-03-24T23:18:48.921 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-24T23:18:48.921 INFO:teuthology.orchestra.run.smithi196.stdout: automake-1.16.1-7.el8.noarch 2023-03-24T23:18:48.921 INFO:teuthology.orchestra.run.smithi196.stdout: bison-3.0.4-10.el8.x86_64 2023-03-24T23:18:48.921 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.921 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.922 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.922 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.922 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.922 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.922 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.922 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.922 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.923 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.923 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.923 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.923 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.923 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.923 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.923 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.923 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.924 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.924 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.924 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.924 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.924 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.924 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:48.924 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-24T23:18:48.924 INFO:teuthology.orchestra.run.smithi196.stdout: flex-2.6.1-9.el8.x86_64 2023-03-24T23:18:48.925 INFO:teuthology.orchestra.run.smithi196.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-24T23:18:48.925 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-24T23:18:48.925 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-3.el8.x86_64 2023-03-24T23:18:48.925 INFO:teuthology.orchestra.run.smithi196.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-24T23:18:48.925 INFO:teuthology.orchestra.run.smithi196.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-24T23:18:48.925 INFO:teuthology.orchestra.run.smithi196.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-24T23:18:48.925 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.926 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.926 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.926 INFO:teuthology.orchestra.run.smithi196.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-24T23:18:48.926 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-24T23:18:48.926 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-24T23:18:48.926 INFO:teuthology.orchestra.run.smithi196.stdout: libpq-13.5-1.el8.x86_64 2023-03-24T23:18:48.926 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-24T23:18:48.926 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-24T23:18:48.927 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.927 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.927 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-24T23:18:48.927 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.927 INFO:teuthology.orchestra.run.smithi196.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-24T23:18:48.927 INFO:teuthology.orchestra.run.smithi196.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-24T23:18:48.927 INFO:teuthology.orchestra.run.smithi196.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-24T23:18:48.927 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-24T23:18:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-24T23:18:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-24T23:18:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-24T23:18:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-24T23:18:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-24T23:18:48.928 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-24T23:18:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-24T23:18:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-24T23:18:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-24T23:18:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-24T23:18:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-24T23:18:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-24T23:18:48.929 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-24T23:18:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-24T23:18:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-24T23:18:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-24T23:18:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-24T23:18:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-24T23:18:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.930 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.931 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-24T23:18:48.931 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-24T23:18:48.931 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-24T23:18:48.931 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-24T23:18:48.931 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-24T23:18:48.931 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-24T23:18:48.931 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-24T23:18:48.932 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-24T23:18:48.932 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-24T23:18:48.932 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-24T23:18:48.932 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-24T23:18:48.932 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-24T23:18:48.932 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-24T23:18:48.932 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-24T23:18:48.932 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-24T23:18:48.933 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-24T23:18:48.933 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-24T23:18:48.933 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-24T23:18:48.933 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-24T23:18:48.933 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-24T23:18:48.933 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-24T23:18:48.933 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-24T23:18:48.933 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-2.6-1.el8.noarch 2023-03-24T23:18:48.934 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-24T23:18:48.934 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-24T23:18:48.934 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-24T23:18:48.934 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-24T23:18:48.934 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-24T23:18:48.934 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-24T23:18:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-24T23:18:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-24T23:18:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-24T23:18:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-24T23:18:48.935 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-24T23:18:48.936 INFO:teuthology.orchestra.run.smithi196.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-24T23:18:48.936 INFO:teuthology.orchestra.run.smithi196.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-24T23:18:48.936 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-24T23:18:48.936 INFO:teuthology.orchestra.run.smithi196.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-24T23:18:48.936 INFO:teuthology.orchestra.run.smithi196.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-24T23:18:48.936 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-24T23:18:48.936 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-24T23:18:48.937 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-24T23:18:48.937 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-24T23:18:48.937 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-24T23:18:48.937 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.937 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.937 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:48.937 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-24T23:18:48.938 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-24T23:18:48.938 INFO:teuthology.orchestra.run.smithi196.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-24T23:18:48.938 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-24T23:18:48.938 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-24T23:18:48.938 INFO:teuthology.orchestra.run.smithi196.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-24T23:18:48.939 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:18:48.939 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2023-03-24T23:18:49.061 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : bison-3.0.4-10.el8.x86_64 129/134 2023-03-24T23:18:49.073 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: bison-3.0.4-10.el8.x86_64 129/134 2023-03-24T23:18:49.104 DEBUG:teuthology.parallel:result is None 2023-03-24T23:18:49.220 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-24T23:18:49.227 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: NetworkManager-initscripts-updown-1:1.36.0-9.el8 130/134 2023-03-24T23:18:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: Installing : libaio-devel-0.3.112-1.el8.x86_64 131/134 2023-03-24T23:18:49.405 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-24T23:18:49.606 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-24T23:18:49.606 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-24T23:18:49.867 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 133/134 2023-03-24T23:18:50.051 INFO:teuthology.orchestra.run.smithi191.stdout: Cleanup : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-24T23:18:50.364 INFO:teuthology.orchestra.run.smithi191.stdout: Running scriptlet: iproute-5.15.0-4.el8.x86_64 134/134 2023-03-24T23:18:50.364 INFO:teuthology.orchestra.run.smithi191.stdout:[/usr/lib/tmpfiles.d/journal-nocow.conf:26] Failed to resolve specifier: uninitialized /etc detected, skipping 2023-03-24T23:18:50.364 INFO:teuthology.orchestra.run.smithi191.stdout:All rules containing unresolvable specifiers will be skipped. 2023-03-24T23:18:50.364 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:18:50.580 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 1/134 2023-03-24T23:18:50.581 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2/134 2023-03-24T23:18:50.581 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 3/134 2023-03-24T23:18:50.581 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 4/134 2023-03-24T23:18:50.581 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-3124.gc19e8 5/134 2023-03-24T23:18:50.581 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 6/134 2023-03-24T23:18:50.582 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 7/134 2023-03-24T23:18:50.582 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 8/134 2023-03-24T23:18:50.582 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 9/134 2023-03-24T23:18:50.582 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 10/134 2023-03-24T23:18:50.582 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 11/134 2023-03-24T23:18:50.582 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 12/134 2023-03-24T23:18:50.582 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 13/134 2023-03-24T23:18:50.583 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_ 14/134 2023-03-24T23:18:50.583 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 15/134 2023-03-24T23:18:50.583 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 16/134 2023-03-24T23:18:50.583 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_6 17/134 2023-03-24T23:18:50.583 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86 18/134 2023-03-24T23:18:50.583 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 19/134 2023-03-24T23:18:50.583 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el 20/134 2023-03-24T23:18:50.583 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8. 21/134 2023-03-24T23:18:50.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_6 22/134 2023-03-24T23:18:50.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 23/134 2023-03-24T23:18:50.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 24/134 2023-03-24T23:18:50.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 25/134 2023-03-24T23:18:50.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 26/134 2023-03-24T23:18:50.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 27/134 2023-03-24T23:18:50.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 28/134 2023-03-24T23:18:50.584 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a. 29/134 2023-03-24T23:18:50.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noa 30/134 2023-03-24T23:18:50.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.n 31/134 2023-03-24T23:18:50.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19 32/134 2023-03-24T23:18:50.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el 33/134 2023-03-24T23:18:50.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 34/134 2023-03-24T23:18:50.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.e 35/134 2023-03-24T23:18:50.585 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 36/134 2023-03-24T23:18:50.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 37/134 2023-03-24T23:18:50.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 38/134 2023-03-24T23:18:50.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 39/134 2023-03-24T23:18:50.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 40/134 2023-03-24T23:18:50.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 41/134 2023-03-24T23:18:50.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 42/134 2023-03-24T23:18:50.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 43/134 2023-03-24T23:18:50.586 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 44/134 2023-03-24T23:18:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 45/134 2023-03-24T23:18:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 46/134 2023-03-24T23:18:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 47/134 2023-03-24T23:18:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 48/134 2023-03-24T23:18:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 49/134 2023-03-24T23:18:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 50/134 2023-03-24T23:18:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 51/134 2023-03-24T23:18:50.587 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 52/134 2023-03-24T23:18:50.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 53/134 2023-03-24T23:18:50.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 54/134 2023-03-24T23:18:50.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 55/134 2023-03-24T23:18:50.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 56/134 2023-03-24T23:18:50.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-portend-2.6-1.el8.noarch 57/134 2023-03-24T23:18:50.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 58/134 2023-03-24T23:18:50.588 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 59/134 2023-03-24T23:18:50.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-rsa-4.8-1.el8.noarch 60/134 2023-03-24T23:18:50.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 61/134 2023-03-24T23:18:50.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 62/134 2023-03-24T23:18:50.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 63/134 2023-03-24T23:18:50.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 64/134 2023-03-24T23:18:50.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 65/134 2023-03-24T23:18:50.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 66/134 2023-03-24T23:18:50.589 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 67/134 2023-03-24T23:18:50.590 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 68/134 2023-03-24T23:18:50.590 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 69/134 2023-03-24T23:18:50.590 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 70/134 2023-03-24T23:18:50.590 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 71/134 2023-03-24T23:18:50.590 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librabbitmq-0.9.0-3.el8.x86_64 72/134 2023-03-24T23:18:50.590 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libselinux-devel-2.9-5.el8.x86_64 73/134 2023-03-24T23:18:50.590 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 74/134 2023-03-24T23:18:50.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-utf32-10.32-2.el8.x86_64 75/134 2023-03-24T23:18:50.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : keyutils-libs-devel-1.5.10-9.el8.x86_64 76/134 2023-03-24T23:18:50.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libsepol-devel-2.9-3.el8.x86_64 77/134 2023-03-24T23:18:50.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 78/134 2023-03-24T23:18:50.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 79/134 2023-03-24T23:18:50.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-utf16-10.32-2.el8.x86_64 80/134 2023-03-24T23:18:50.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : krb5-devel-1.18.2-14.el8.x86_64 81/134 2023-03-24T23:18:50.591 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libaio-devel-0.3.112-1.el8.x86_64 82/134 2023-03-24T23:18:50.592 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 83/134 2023-03-24T23:18:50.592 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cryptography-3.2.1-5.el8.x86_64 84/134 2023-03-24T23:18:50.592 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libverto-devel-0.3.0-5.el8.x86_64 85/134 2023-03-24T23:18:50.592 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : pcre2-devel-10.32-2.el8.x86_64 86/134 2023-03-24T23:18:50.592 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-cffi-1.11.5-5.el8.x86_64 87/134 2023-03-24T23:18:50.592 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openssl-devel-1:1.1.1k-7.el8_6.x86_64 88/134 2023-03-24T23:18:50.592 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : xfsprogs-devel-5.0.0-10.el8.x86_64 89/134 2023-03-24T23:18:50.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : fuse-2.9.7-15.el8.x86_64 90/134 2023-03-24T23:18:50.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : elfutils-libelf-devel-0.186-1.el8.x86_64 91/134 2023-03-24T23:18:50.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libuuid-devel-2.32.1-35.el8.x86_64 92/134 2023-03-24T23:18:50.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : zlib-devel-1.2.11-18.el8_5.x86_64 93/134 2023-03-24T23:18:50.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libcom_err-devel-1.45.6-4.el8.x86_64 94/134 2023-03-24T23:18:50.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libgfortran-8.5.0-10.1.el8_6.x86_64 95/134 2023-03-24T23:18:50.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libquadmath-8.5.0-10.1.el8_6.x86_64 96/134 2023-03-24T23:18:50.593 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : NetworkManager-initscripts-updown-1:1.36.0-9.el8 97/134 2023-03-24T23:18:50.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd 98/134 2023-03-24T23:18:50.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 99/134 2023-03-24T23:18:50.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 100/134 2023-03-24T23:18:50.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 101/134 2023-03-24T23:18:50.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 102/134 2023-03-24T23:18:50.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 103/134 2023-03-24T23:18:50.594 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : bison-3.0.4-10.el8.x86_64 104/134 2023-03-24T23:18:50.595 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 105/134 2023-03-24T23:18:50.595 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 106/134 2023-03-24T23:18:50.595 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 107/134 2023-03-24T23:18:50.595 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : automake-1.16.1-7.el8.noarch 108/134 2023-03-24T23:18:50.595 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 109/134 2023-03-24T23:18:50.595 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 110/134 2023-03-24T23:18:50.595 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : flex-2.6.1-9.el8.x86_64 111/134 2023-03-24T23:18:50.595 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 112/134 2023-03-24T23:18:50.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 113/134 2023-03-24T23:18:50.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 114/134 2023-03-24T23:18:50.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 115/134 2023-03-24T23:18:50.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : uuid-1.6.2-43.el8.x86_64 116/134 2023-03-24T23:18:50.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 117/134 2023-03-24T23:18:50.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 118/134 2023-03-24T23:18:50.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 119/134 2023-03-24T23:18:50.596 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libtool-2.4.6-25.el8.x86_64 120/134 2023-03-24T23:18:50.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-threads-0.3.15-3.el8.x86_64 121/134 2023-03-24T23:18:50.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : libpq-13.5-1.el8.x86_64 122/134 2023-03-24T23:18:50.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : openblas-0.3.15-3.el8.x86_64 123/134 2023-03-24T23:18:50.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : jq-1.6-3.el8.x86_64 124/134 2023-03-24T23:18:50.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 125/134 2023-03-24T23:18:50.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-server-10.21-2.module+el8.6.0+15342+5 126/134 2023-03-24T23:18:50.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-contrib-10.21-2.module+el8.6.0+15342+ 127/134 2023-03-24T23:18:50.597 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : postgresql-10.21-2.module+el8.6.0+15342+53518fac 128/134 2023-03-24T23:18:50.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 129/134 2023-03-24T23:18:50.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 130/134 2023-03-24T23:18:50.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 131/134 2023-03-24T23:18:50.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 132/134 2023-03-24T23:18:50.598 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : iproute-5.15.0-4.el8_6.1.x86_64 133/134 2023-03-24T23:18:53.335 INFO:teuthology.orchestra.run.smithi191.stdout: Verifying : iproute-5.15.0-4.el8.x86_64 134/134 2023-03-24T23:18:53.335 INFO:teuthology.orchestra.run.smithi191.stdout:Last metadata expiration check: 0:01:52 ago on Fri 24 Mar 2023 11:17:01 PM UTC. 2023-03-24T23:18:55.985 INFO:teuthology.orchestra.run.smithi191.stdout:Installed products updated. 2023-03-24T23:18:56.205 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:18:56.205 INFO:teuthology.orchestra.run.smithi191.stdout:Upgraded: 2023-03-24T23:18:56.205 INFO:teuthology.orchestra.run.smithi191.stdout: iproute-5.15.0-4.el8_6.1.x86_64 2023-03-24T23:18:56.206 INFO:teuthology.orchestra.run.smithi191.stdout: librados2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.206 INFO:teuthology.orchestra.run.smithi191.stdout: librbd1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.206 INFO:teuthology.orchestra.run.smithi191.stdout:Installed: 2023-03-24T23:18:56.206 INFO:teuthology.orchestra.run.smithi191.stdout: NetworkManager-initscripts-updown-1:1.36.0-9.el8_6.noarch 2023-03-24T23:18:56.206 INFO:teuthology.orchestra.run.smithi191.stdout: automake-1.16.1-7.el8.noarch 2023-03-24T23:18:56.206 INFO:teuthology.orchestra.run.smithi191.stdout: bison-3.0.4-10.el8.x86_64 2023-03-24T23:18:56.206 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.207 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-base-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.207 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.207 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.207 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-grafana-dashboards-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.207 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-immutable-object-cache-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.207 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mds-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.207 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-dashboard-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-diskprediction-local-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-modules-core-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mgr-rook-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-mon-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-osd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.208 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-prometheus-alerts-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.209 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-radosgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.209 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-selinux-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.209 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-test-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.209 INFO:teuthology.orchestra.run.smithi191.stdout: ceph-volume-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.209 INFO:teuthology.orchestra.run.smithi191.stdout: cephadm-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.209 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.209 INFO:teuthology.orchestra.run.smithi191.stdout: cephfs-top-2:18.0.0-3124.gc19e8f2a.el8.noarch 2023-03-24T23:18:56.210 INFO:teuthology.orchestra.run.smithi191.stdout: elfutils-libelf-devel-0.186-1.el8.x86_64 2023-03-24T23:18:56.210 INFO:teuthology.orchestra.run.smithi191.stdout: flex-2.6.1-9.el8.x86_64 2023-03-24T23:18:56.210 INFO:teuthology.orchestra.run.smithi191.stdout: fuse-2.9.7-15.el8.x86_64 2023-03-24T23:18:56.210 INFO:teuthology.orchestra.run.smithi191.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-03-24T23:18:56.210 INFO:teuthology.orchestra.run.smithi191.stdout: jq-1.6-3.el8.x86_64 2023-03-24T23:18:56.210 INFO:teuthology.orchestra.run.smithi191.stdout: keyutils-libs-devel-1.5.10-9.el8.x86_64 2023-03-24T23:18:56.210 INFO:teuthology.orchestra.run.smithi191.stdout: krb5-devel-1.18.2-14.el8.x86_64 2023-03-24T23:18:56.210 INFO:teuthology.orchestra.run.smithi191.stdout: libaio-devel-0.3.112-1.el8.x86_64 2023-03-24T23:18:56.211 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.211 INFO:teuthology.orchestra.run.smithi191.stdout: libcephfs2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.211 INFO:teuthology.orchestra.run.smithi191.stdout: libcephsqlite-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.211 INFO:teuthology.orchestra.run.smithi191.stdout: libcom_err-devel-1.45.6-4.el8.x86_64 2023-03-24T23:18:56.211 INFO:teuthology.orchestra.run.smithi191.stdout: libgfortran-8.5.0-10.1.el8_6.x86_64 2023-03-24T23:18:56.211 INFO:teuthology.orchestra.run.smithi191.stdout: liboath-2.6.2-3.el8.x86_64 2023-03-24T23:18:56.211 INFO:teuthology.orchestra.run.smithi191.stdout: libpq-13.5-1.el8.x86_64 2023-03-24T23:18:56.212 INFO:teuthology.orchestra.run.smithi191.stdout: libquadmath-8.5.0-10.1.el8_6.x86_64 2023-03-24T23:18:56.212 INFO:teuthology.orchestra.run.smithi191.stdout: librabbitmq-0.9.0-3.el8.x86_64 2023-03-24T23:18:56.212 INFO:teuthology.orchestra.run.smithi191.stdout: librados-devel-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.212 INFO:teuthology.orchestra.run.smithi191.stdout: libradosstriper1-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.212 INFO:teuthology.orchestra.run.smithi191.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-03-24T23:18:56.212 INFO:teuthology.orchestra.run.smithi191.stdout: librgw2-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.212 INFO:teuthology.orchestra.run.smithi191.stdout: libselinux-devel-2.9-5.el8.x86_64 2023-03-24T23:18:56.212 INFO:teuthology.orchestra.run.smithi191.stdout: libsepol-devel-2.9-3.el8.x86_64 2023-03-24T23:18:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: libtool-2.4.6-25.el8.x86_64 2023-03-24T23:18:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: libunwind-1.3.1-3.el8.x86_64 2023-03-24T23:18:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: libuuid-devel-2.32.1-35.el8.x86_64 2023-03-24T23:18:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: libverto-devel-0.3.0-5.el8.x86_64 2023-03-24T23:18:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-03-24T23:18:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-03-24T23:18:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-0.3.15-3.el8.x86_64 2023-03-24T23:18:56.213 INFO:teuthology.orchestra.run.smithi191.stdout: openblas-threads-0.3.15-3.el8.x86_64 2023-03-24T23:18:56.214 INFO:teuthology.orchestra.run.smithi191.stdout: openssl-devel-1:1.1.1k-7.el8_6.x86_64 2023-03-24T23:18:56.214 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-devel-10.32-2.el8.x86_64 2023-03-24T23:18:56.214 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf16-10.32-2.el8.x86_64 2023-03-24T23:18:56.214 INFO:teuthology.orchestra.run.smithi191.stdout: pcre2-utf32-10.32-2.el8.x86_64 2023-03-24T23:18:56.214 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-24T23:18:56.214 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-contrib-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-24T23:18:56.214 INFO:teuthology.orchestra.run.smithi191.stdout: postgresql-server-10.21-2.module+el8.6.0+15342+53518fac.x86_64 2023-03-24T23:18:56.215 INFO:teuthology.orchestra.run.smithi191.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-03-24T23:18:56.215 INFO:teuthology.orchestra.run.smithi191.stdout: python3-babel-2.5.1-7.el8.noarch 2023-03-24T23:18:56.215 INFO:teuthology.orchestra.run.smithi191.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-03-24T23:18:56.215 INFO:teuthology.orchestra.run.smithi191.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-03-24T23:18:56.215 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-03-24T23:18:56.215 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-argparse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.215 INFO:teuthology.orchestra.run.smithi191.stdout: python3-ceph-common-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.215 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cephfs-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.216 INFO:teuthology.orchestra.run.smithi191.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-03-24T23:18:56.216 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cffi-1.11.5-5.el8.x86_64 2023-03-24T23:18:56.216 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-03-24T23:18:56.216 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-03-24T23:18:56.216 INFO:teuthology.orchestra.run.smithi191.stdout: python3-cryptography-3.2.1-5.el8.x86_64 2023-03-24T23:18:56.216 INFO:teuthology.orchestra.run.smithi191.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-03-24T23:18:56.216 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-6.2-6.el8.noarch 2023-03-24T23:18:56.216 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-03-24T23:18:56.217 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-03-24T23:18:56.217 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-03-24T23:18:56.217 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-03-24T23:18:56.217 INFO:teuthology.orchestra.run.smithi191.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-03-24T23:18:56.217 INFO:teuthology.orchestra.run.smithi191.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-03-24T23:18:56.217 INFO:teuthology.orchestra.run.smithi191.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-03-24T23:18:56.217 INFO:teuthology.orchestra.run.smithi191.stdout: python3-mako-1.0.6-13.el8.noarch 2023-03-24T23:18:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-03-24T23:18:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-03-24T23:18:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-03-24T23:18:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-03-24T23:18:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-03-24T23:18:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-03-24T23:18:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-03-24T23:18:56.218 INFO:teuthology.orchestra.run.smithi191.stdout: python3-portend-2.6-1.el8.noarch 2023-03-24T23:18:56.219 INFO:teuthology.orchestra.run.smithi191.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-03-24T23:18:56.219 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-03-24T23:18:56.219 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-03-24T23:18:56.219 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-03-24T23:18:56.219 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pycparser-2.14-14.el8.noarch 2023-03-24T23:18:56.219 INFO:teuthology.orchestra.run.smithi191.stdout: python3-pytz-2017.2-9.el8.noarch 2023-03-24T23:18:56.219 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rados-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.219 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.220 INFO:teuthology.orchestra.run.smithi191.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-03-24T23:18:56.220 INFO:teuthology.orchestra.run.smithi191.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-03-24T23:18:56.220 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rgw-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.220 INFO:teuthology.orchestra.run.smithi191.stdout: python3-routes-2.4.1-12.el8.noarch 2023-03-24T23:18:56.220 INFO:teuthology.orchestra.run.smithi191.stdout: python3-rsa-4.8-1.el8.noarch 2023-03-24T23:18:56.220 INFO:teuthology.orchestra.run.smithi191.stdout: python3-scipy-1.0.0-21.module+el8.5.0+10916+41bd434d.x86_64 2023-03-24T23:18:56.220 INFO:teuthology.orchestra.run.smithi191.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-03-24T23:18:56.221 INFO:teuthology.orchestra.run.smithi191.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-03-24T23:18:56.221 INFO:teuthology.orchestra.run.smithi191.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-03-24T23:18:56.221 INFO:teuthology.orchestra.run.smithi191.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-03-24T23:18:56.221 INFO:teuthology.orchestra.run.smithi191.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2023-03-24T23:18:56.221 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-03-24T23:18:56.221 INFO:teuthology.orchestra.run.smithi191.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-03-24T23:18:56.221 INFO:teuthology.orchestra.run.smithi191.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-03-24T23:18:56.221 INFO:teuthology.orchestra.run.smithi191.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-03-24T23:18:56.222 INFO:teuthology.orchestra.run.smithi191.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-03-24T23:18:56.222 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-fuse-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.222 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-mirror-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.222 INFO:teuthology.orchestra.run.smithi191.stdout: rbd-nbd-2:18.0.0-3124.gc19e8f2a.el8.x86_64 2023-03-24T23:18:56.222 INFO:teuthology.orchestra.run.smithi191.stdout: socat-1.7.4.1-1.el8.x86_64 2023-03-24T23:18:56.222 INFO:teuthology.orchestra.run.smithi191.stdout: thrift-0.13.0-2.el8.x86_64 2023-03-24T23:18:56.222 INFO:teuthology.orchestra.run.smithi191.stdout: uuid-1.6.2-43.el8.x86_64 2023-03-24T23:18:56.222 INFO:teuthology.orchestra.run.smithi191.stdout: xfsprogs-devel-5.0.0-10.el8.x86_64 2023-03-24T23:18:56.223 INFO:teuthology.orchestra.run.smithi191.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-03-24T23:18:56.223 INFO:teuthology.orchestra.run.smithi191.stdout: zlib-devel-1.2.11-18.el8_5.x86_64 2023-03-24T23:18:56.223 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:18:56.223 INFO:teuthology.orchestra.run.smithi191.stdout:Complete! 2023-03-24T23:18:56.464 DEBUG:teuthology.parallel:result is None 2023-03-24T23:18:56.465 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-24T23:18:56.598 DEBUG:teuthology.orchestra.run.smithi191:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-24T23:18:56.620 INFO:teuthology.orchestra.run.smithi191.stdout:18.0.0-3124.gc19e8f2a.el8 2023-03-24T23:18:56.621 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3124.gc19e8f2a.el8 2023-03-24T23:18:56.621 INFO:teuthology.task.install:The correct ceph version 18.0.0-3124.gc19e8f2a is installed. 2023-03-24T23:18:56.622 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=c19e8f2a497e95f75abc43ec14bc6095e30dd2cb 2023-03-24T23:18:56.752 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-03-24T23:18:56.780 INFO:teuthology.orchestra.run.smithi196.stdout:18.0.0-3124.gc19e8f2a.el8 2023-03-24T23:18:56.780 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-3124.gc19e8f2a.el8 2023-03-24T23:18:56.781 INFO:teuthology.task.install:The correct ceph version 18.0.0-3124.gc19e8f2a is installed. 2023-03-24T23:18:56.781 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-03-24T23:18:56.782 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:18:56.782 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-24T23:18:56.820 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:18:56.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-03-24T23:18:56.867 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-03-24T23:18:56.867 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:18:56.868 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/daemon-helper 2023-03-24T23:18:56.893 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-24T23:18:56.959 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:18:56.959 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2023-03-24T23:18:56.989 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-03-24T23:18:57.057 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-03-24T23:18:57.057 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:18:57.058 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-24T23:18:57.083 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-24T23:18:57.149 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:18:57.149 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2023-03-24T23:18:57.177 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-03-24T23:18:57.245 INFO:teuthology.run_tasks:Running task ceph... 2023-03-24T23:18:57.316 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-03-24T23:18:57.317 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 777 /var/log/ceph 2023-03-24T23:18:57.320 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2023-03-24T23:18:57.346 INFO:tasks.ceph:Disabling ceph logrotate... 2023-03-24T23:18:57.347 DEBUG:teuthology.orchestra.run.smithi191:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-24T23:18:57.385 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2023-03-24T23:18:57.413 INFO:tasks.ceph:Creating extra log directories... 2023-03-24T23:18:57.414 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-24T23:18:57.451 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-03-24T23:18:57.513 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-03-24T23:18:57.514 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-03-24T23:18:57.514 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-03-24T23:18:57.514 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:18:57.515 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-24T23:18:57.531 DEBUG:teuthology.orchestra.run.smithi191:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-24T23:18:57.675 DEBUG:teuthology.orchestra.run.smithi191:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-24T23:18:57.701 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:18:57.701 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-03-24T23:18:57.718 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-03-24T23:18:58.151 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-03-24T23:18:58.185 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-03-24T23:18:58.185 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-03-24T23:18:58.186 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi/7219622', 'branch': 'wip-pdonnell-testing-20230323.162417', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_8} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/stock/{k-stock rhel_8} ms-die-on-skipped}} objectstore/bluestore-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/forward-scrub}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7219622', 'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'pdonnell-2023-03-24_22:00:37-fs-wip-pdonnell-testing-20230323.162417-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'rhel', 'os_version': '8.6', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20230323.162417'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd']}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'inode wrongly marked free', 'bad backtrace on inode', 'inode table repaired for inode', 'Scrub error on inode', 'Scrub error on dir', 'Metadata damage detected'], 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'}}, 'kclient': {'syntax': 'v1'}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20230323.162417', 'sha1': '568504667cd0ede116ce6b46562b9a2881100488'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 55, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20230323.162417', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_568504667cd0ede116ce6b46562b9a2881100488/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph.git', 'suite_sha1': '568504667cd0ede116ce6b46562b9a2881100488', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi191.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCpUxo5ay/ucdkX9rleNMm8Ofp24XRXUtJPOK3Ji3XBpTiUgmmwneaY6316+3CG+k4K0p3IebanuyTvZApjm02/dNCvu8yD3rjEpWVgIuqYH8O3lEdN9DXcCX+7xHJE0m18+ZPsJQVsNrNJxY+5zqyNnBc8jpZq2LStsWwPOKPWhauANRkOBFSGDoLppWZbLx/yJs+R0qoNLLK9LpKQTUnqsapXvbD27OpPahzj/qbNxFPPyY6XrrldZ6qEO+EjJOujFLH3gzv0kJu10DCuzxrpz6ston4RPi1O3GgcHnF4Mqr/W9VwZ94UTMh+k4dIFRVqJEKrcwRHa5kFsvH/WhVx0Ygr/MiV/Q86mmV7JT+/MpzIRTtWyei5L9IBeZIE9cwzj+mOk6SD/t1roIVEGVV/uF2AyFDVBPr6F8oZ822OGpUyyaDsWTitPNdTX4TC3zdSrLc9tyDa1P74eH/ZLeTSf+C6pM/cepOhQaa6XzpDiPSuJUm9HOQxUlEAvcqa7Ac=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLVvN3+LmzPZ4ixiwyCkS3KZLZ9R6MOPm+kPtaJbl8fExpzbk8TudXCcJWHBL7DGPWFvye2jT6sei9eiX0EUQYs='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'sha1': 'distro'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c19e8f2a497e95f75abc43ec14bc6095e30dd2cb'}}, {'ceph': None}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_forward_scrub']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'parallel-gzip', 'teuthology_sha1': 'fe9b6e52013911547168fef4a8506325b6c76139', 'timestamp': '2023-03-24_22:00:37', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.4538'} 2023-03-24T23:18:58.187 DEBUG:teuthology.orchestra.run.smithi191:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-24T23:18:58.189 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-03-24T23:18:58.237 DEBUG:teuthology.orchestra.run.smithi191:> sudo install -d -m0777 -- /var/run/ceph 2023-03-24T23:18:58.246 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2023-03-24T23:18:58.305 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:18:58.306 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/scratch_devs of=/dev/stdout 2023-03-24T23:18:58.328 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-24T23:18:58.328 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_1 2023-03-24T23:18:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-24T23:18:58.385 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-24T23:18:58.385 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 359260 Links: 1 2023-03-24T23:18:58.385 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-24T23:18:58.386 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-03-24T23:18:58.386 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-03-24 23:18:50.286106539 +0000 2023-03-24T23:18:58.386 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-03-24 23:16:15.697404229 +0000 2023-03-24T23:18:58.386 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-03-24 23:16:15.697404229 +0000 2023-03-24T23:18:58.386 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-03-24T23:18:58.386 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-24T23:18:58.450 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-03-24T23:18:58.450 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-03-24T23:18:58.450 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000247601 s, 2.1 MB/s 2023-03-24T23:18:58.451 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-24T23:18:58.509 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_2 2023-03-24T23:18:58.566 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-24T23:18:58.566 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-24T23:18:58.567 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 359304 Links: 1 2023-03-24T23:18:58.567 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-24T23:18:58.567 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-03-24T23:18:58.567 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-03-24 23:18:50.286106539 +0000 2023-03-24T23:18:58.567 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-03-24 23:16:16.212395271 +0000 2023-03-24T23:18:58.567 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-03-24 23:16:16.212395271 +0000 2023-03-24T23:18:58.567 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-03-24T23:18:58.568 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-24T23:18:58.631 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-03-24T23:18:58.631 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-03-24T23:18:58.631 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000278016 s, 1.8 MB/s 2023-03-24T23:18:58.632 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-24T23:18:58.690 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_3 2023-03-24T23:18:58.748 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-24T23:18:58.748 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-24T23:18:58.748 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 359367 Links: 1 2023-03-24T23:18:58.748 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-24T23:18:58.748 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-03-24T23:18:58.749 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-03-24 23:18:50.286106539 +0000 2023-03-24T23:18:58.749 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-03-24 23:16:16.707386662 +0000 2023-03-24T23:18:58.749 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-03-24 23:16:16.707386662 +0000 2023-03-24T23:18:58.749 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-03-24T23:18:58.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-24T23:18:58.813 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-03-24T23:18:58.813 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-03-24T23:18:58.813 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.00021275 s, 2.4 MB/s 2023-03-24T23:18:58.814 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-24T23:18:58.871 DEBUG:teuthology.orchestra.run.smithi191:> stat /dev/vg_nvme/lv_4 2023-03-24T23:18:58.928 INFO:teuthology.orchestra.run.smithi191.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-24T23:18:58.929 INFO:teuthology.orchestra.run.smithi191.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-24T23:18:58.929 INFO:teuthology.orchestra.run.smithi191.stdout:Device: 6h/6d Inode: 365872 Links: 1 2023-03-24T23:18:58.929 INFO:teuthology.orchestra.run.smithi191.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-24T23:18:58.929 INFO:teuthology.orchestra.run.smithi191.stdout:Context: system_u:object_r:device_t:s0 2023-03-24T23:18:58.929 INFO:teuthology.orchestra.run.smithi191.stdout:Access: 2023-03-24 23:18:50.286106539 +0000 2023-03-24T23:18:58.929 INFO:teuthology.orchestra.run.smithi191.stdout:Modify: 2023-03-24 23:16:17.226377635 +0000 2023-03-24T23:18:58.930 INFO:teuthology.orchestra.run.smithi191.stdout:Change: 2023-03-24 23:16:17.226377635 +0000 2023-03-24T23:18:58.930 INFO:teuthology.orchestra.run.smithi191.stdout: Birth: - 2023-03-24T23:18:58.930 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-24T23:18:58.993 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records in 2023-03-24T23:18:58.993 INFO:teuthology.orchestra.run.smithi191.stderr:1+0 records out 2023-03-24T23:18:58.993 INFO:teuthology.orchestra.run.smithi191.stderr:512 bytes copied, 0.000243523 s, 2.1 MB/s 2023-03-24T23:18:58.994 DEBUG:teuthology.orchestra.run.smithi191:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-24T23:18:59.051 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-24T23:18:59.051 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:18:59.052 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2023-03-24T23:18:59.072 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-03-24T23:18:59.072 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2023-03-24T23:18:59.132 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-03-24T23:18:59.132 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-24T23:18:59.132 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 360123 Links: 1 2023-03-24T23:18:59.132 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-24T23:18:59.132 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2023-03-24T23:18:59.133 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2023-03-24 23:18:42.879514724 +0000 2023-03-24T23:18:59.133 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2023-03-24 23:16:15.702612525 +0000 2023-03-24T23:18:59.133 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2023-03-24 23:16:15.702612525 +0000 2023-03-24T23:18:59.133 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2023-03-24T23:18:59.133 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-03-24T23:18:59.202 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2023-03-24T23:18:59.202 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2023-03-24T23:18:59.202 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000290616 s, 1.8 MB/s 2023-03-24T23:18:59.203 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-03-24T23:18:59.264 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2023-03-24T23:18:59.322 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-03-24T23:18:59.322 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-24T23:18:59.322 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 360209 Links: 1 2023-03-24T23:18:59.323 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-24T23:18:59.323 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2023-03-24T23:18:59.323 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2023-03-24 23:18:42.879514724 +0000 2023-03-24T23:18:59.323 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2023-03-24 23:16:16.248600483 +0000 2023-03-24T23:18:59.323 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2023-03-24 23:16:16.248600483 +0000 2023-03-24T23:18:59.323 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2023-03-24T23:18:59.324 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-03-24T23:18:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2023-03-24T23:18:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2023-03-24T23:18:59.389 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000209193 s, 2.4 MB/s 2023-03-24T23:18:59.390 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-03-24T23:18:59.447 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2023-03-24T23:18:59.504 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-03-24T23:18:59.505 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-24T23:18:59.505 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 367744 Links: 1 2023-03-24T23:18:59.505 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-24T23:18:59.505 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2023-03-24T23:18:59.505 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2023-03-24 23:18:42.880514702 +0000 2023-03-24T23:18:59.506 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2023-03-24 23:16:16.769588992 +0000 2023-03-24T23:18:59.506 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2023-03-24 23:16:16.769588992 +0000 2023-03-24T23:18:59.506 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2023-03-24T23:18:59.506 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-03-24T23:18:59.569 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2023-03-24T23:18:59.569 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2023-03-24T23:18:59.570 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.0002056 s, 2.5 MB/s 2023-03-24T23:18:59.570 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-03-24T23:18:59.628 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2023-03-24T23:18:59.683 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-03-24T23:18:59.684 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-03-24T23:18:59.684 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 6h/6d Inode: 360286 Links: 1 2023-03-24T23:18:59.684 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-03-24T23:18:59.684 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2023-03-24T23:18:59.684 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2023-03-24 23:18:42.880514702 +0000 2023-03-24T23:18:59.685 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2023-03-24 23:16:17.299577303 +0000 2023-03-24T23:18:59.685 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2023-03-24 23:16:17.299577303 +0000 2023-03-24T23:18:59.685 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: - 2023-03-24T23:18:59.685 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-03-24T23:18:59.748 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2023-03-24T23:18:59.748 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2023-03-24T23:18:59.748 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000192798 s, 2.7 MB/s 2023-03-24T23:18:59.749 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-03-24T23:18:59.806 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-24T23:18:59.807 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-24T23:18:59.807 INFO:tasks.ceph:Generating config... 2023-03-24T23:18:59.808 INFO:tasks.ceph:[client] client mount timeout = 600 2023-03-24T23:18:59.809 INFO:tasks.ceph:[client] debug client = 20 2023-03-24T23:18:59.809 INFO:tasks.ceph:[client] debug ms = 1 2023-03-24T23:18:59.809 INFO:tasks.ceph:[client] pid file = 2023-03-24T23:18:59.809 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-03-24T23:18:59.809 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-03-24T23:18:59.810 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-03-24T23:18:59.810 INFO:tasks.ceph:[mds] debug mds = 20 2023-03-24T23:18:59.810 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-03-24T23:18:59.810 INFO:tasks.ceph:[mds] debug ms = 1 2023-03-24T23:18:59.810 INFO:tasks.ceph:[mds] mds debug frag = True 2023-03-24T23:18:59.811 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-03-24T23:18:59.811 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-03-24T23:18:59.811 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-03-24T23:18:59.811 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-03-24T23:18:59.811 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-03-24T23:18:59.812 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-03-24T23:18:59.812 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-03-24T23:18:59.812 INFO:tasks.ceph:[mgr] debug ms = 1 2023-03-24T23:18:59.812 INFO:tasks.ceph:[mon] debug mon = 20 2023-03-24T23:18:59.812 INFO:tasks.ceph:[mon] debug ms = 1 2023-03-24T23:18:59.813 INFO:tasks.ceph:[mon] debug paxos = 20 2023-03-24T23:18:59.813 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-03-24T23:18:59.813 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-03-24T23:18:59.813 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-03-24T23:18:59.813 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-03-24T23:18:59.813 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-03-24T23:18:59.814 INFO:tasks.ceph:[osd] debug ms = 1 2023-03-24T23:18:59.814 INFO:tasks.ceph:[osd] debug osd = 20 2023-03-24T23:18:59.814 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-03-24T23:18:59.814 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-03-24T23:18:59.814 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-03-24T23:18:59.815 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-03-24T23:18:59.815 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-03-24T23:18:59.815 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-03-24T23:18:59.815 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-03-24T23:18:59.815 INFO:tasks.ceph:Setting up mon.a... 2023-03-24T23:18:59.816 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-03-24T23:18:59.880 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.keyring 2023-03-24T23:18:59.883 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-03-24T23:18:59.931 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-24T23:18:59.958 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.191'), ('mon.b', '172.21.15.196'), ('mon.c', '[v2:172.21.15.196:3301,v1:172.21.15.196:6790]')] 2023-03-24T23:18:59.959 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.191,172.21.15.196,[v2:172.21.15.196:3301,v1:172.21.15.196:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'false', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-03-24T23:18:59.959 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:18:59.960 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-03-24T23:19:00.015 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.191 --add b 172.21.15.196 --addv c '[v2:172.21.15.196:3301,v1:172.21.15.196:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:00.104 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:00.105 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: generated fsid 72b355de-1f4b-4f02-899f-839bc265bc0f 2023-03-24T23:19:00.105 INFO:teuthology.orchestra.run.smithi191.stdout:setting min_mon_release = pacific 2023-03-24T23:19:00.105 INFO:teuthology.orchestra.run.smithi191.stdout:epoch 0 2023-03-24T23:19:00.105 INFO:teuthology.orchestra.run.smithi191.stdout:fsid 72b355de-1f4b-4f02-899f-839bc265bc0f 2023-03-24T23:19:00.105 INFO:teuthology.orchestra.run.smithi191.stdout:last_changed 2023-03-24T23:19:00.105207+0000 2023-03-24T23:19:00.106 INFO:teuthology.orchestra.run.smithi191.stdout:created 2023-03-24T23:19:00.105207+0000 2023-03-24T23:19:00.106 INFO:teuthology.orchestra.run.smithi191.stdout:min_mon_release 16 (pacific) 2023-03-24T23:19:00.106 INFO:teuthology.orchestra.run.smithi191.stdout:election_strategy: 1 2023-03-24T23:19:00.106 INFO:teuthology.orchestra.run.smithi191.stdout:0: [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] mon.a 2023-03-24T23:19:00.106 INFO:teuthology.orchestra.run.smithi191.stdout:1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2023-03-24T23:19:00.106 INFO:teuthology.orchestra.run.smithi191.stdout:2: [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] mon.c 2023-03-24T23:19:00.106 INFO:teuthology.orchestra.run.smithi191.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-03-24T23:19:00.107 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-03-24T23:19:00.123 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 72b355de-1f4b-4f02-899f-839bc265bc0f... 2023-03-24T23:19:00.124 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-24T23:19:00.166 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-03-24T23:19:00.209 INFO:teuthology.orchestra.run.smithi191.stdout:[global] 2023-03-24T23:19:00.209 INFO:teuthology.orchestra.run.smithi191.stdout: chdir = "" 2023-03-24T23:19:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-24T23:19:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: auth supported = cephx 2023-03-24T23:19:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.210 INFO:teuthology.orchestra.run.smithi191.stdout: filestore xattr use omap = true 2023-03-24T23:19:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: mon clock drift allowed = 1.000 2023-03-24T23:19:00.211 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: osd crush chooseleaf type = 0 2023-03-24T23:19:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: auth debug = true 2023-03-24T23:19:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.212 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on old message = true 2023-03-24T23:19:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on bug = true 2023-03-24T23:19:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-24T23:19:00.213 INFO:teuthology.orchestra.run.smithi191.stdout: mon pg warn max object skew = 0 2023-03-24T23:19:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: # disable pg_autoscaler by default for new pools 2023-03-24T23:19:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-24T23:19:00.214 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default size = 2 2023-03-24T23:19:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.215 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow primary affinity = true 2023-03-24T23:19:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd allow pg remap = true 2023-03-24T23:19:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on legacy crush tunables = false 2023-03-24T23:19:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on crush straw calc version zero = false 2023-03-24T23:19:00.216 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on no sortbitwise = false 2023-03-24T23:19:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on osd down out interval zero = false 2023-03-24T23:19:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: mon warn on too few osds = false 2023-03-24T23:19:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-24T23:19:00.217 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-24T23:19:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: mon_allow_pool_size_one = true 2023-03-24T23:19:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-24T23:19:00.218 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: osd default data pool replay window = 5 2023-03-24T23:19:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.219 INFO:teuthology.orchestra.run.smithi191.stdout: mon allow pool delete = true 2023-03-24T23:19:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: mon cluster log file level = debug 2023-03-24T23:19:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: debug asserts on shutdown = true 2023-03-24T23:19:00.220 INFO:teuthology.orchestra.run.smithi191.stdout: mon health detail to clog = false 2023-03-24T23:19:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: mon host = "172.21.15.191,172.21.15.196,[v2:172.21.15.196:3301,v1:172.21.15.196:6790]" 2023-03-24T23:19:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: ms die on skipped message = False 2023-03-24T23:19:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: fsid = 72b355de-1f4b-4f02-899f-839bc265bc0f 2023-03-24T23:19:00.221 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.222 INFO:teuthology.orchestra.run.smithi191.stdout:[osd] 2023-03-24T23:19:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: osd journal size = 100 2023-03-24T23:19:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.222 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub load threshold = 5.0 2023-03-24T23:19:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: osd scrub max interval = 600 2023-03-24T23:19:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.223 INFO:teuthology.orchestra.run.smithi191.stdout: osd recover clone overlap = true 2023-03-24T23:19:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: osd recovery max chunk = 1048576 2023-03-24T23:19:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug shutdown = true 2023-03-24T23:19:00.224 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug op order = false 2023-03-24T23:19:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug verify stray on activate = true 2023-03-24T23:19:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: osd open classes on start = true 2023-03-24T23:19:00.225 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug pg log writeout = true 2023-03-24T23:19:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: osd deep scrub update digest min age = 30 2023-03-24T23:19:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.226 INFO:teuthology.orchestra.run.smithi191.stdout: osd map max advance = 10 2023-03-24T23:19:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: journal zero on create = true 2023-03-24T23:19:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.227 INFO:teuthology.orchestra.run.smithi191.stdout: filestore ondisk finisher threads = 3 2023-03-24T23:19:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: filestore apply finisher threads = 3 2023-03-24T23:19:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: bdev debug aio = true 2023-03-24T23:19:00.228 INFO:teuthology.orchestra.run.smithi191.stdout: osd debug misdirected ops = true 2023-03-24T23:19:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore block size = 96636764160 2023-03-24T23:19:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: bluestore fsck on mount = True 2023-03-24T23:19:00.229 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluefs = 20 2023-03-24T23:19:00.230 INFO:teuthology.orchestra.run.smithi191.stdout: debug bluestore = 20 2023-03-24T23:19:00.230 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-03-24T23:19:00.230 INFO:teuthology.orchestra.run.smithi191.stdout: debug osd = 20 2023-03-24T23:19:00.230 INFO:teuthology.orchestra.run.smithi191.stdout: debug rocksdb = 10 2023-03-24T23:19:00.231 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd backfillfull_ratio = 0.85 2023-03-24T23:19:00.231 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd full ratio = 0.9 2023-03-24T23:19:00.231 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd nearfull ratio = 0.8 2023-03-24T23:19:00.231 INFO:teuthology.orchestra.run.smithi191.stdout: osd failsafe full ratio = 0.95 2023-03-24T23:19:00.232 INFO:teuthology.orchestra.run.smithi191.stdout: osd objectstore = bluestore 2023-03-24T23:19:00.232 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2023-03-24T23:19:00.232 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.232 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr] 2023-03-24T23:19:00.233 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-03-24T23:19:00.233 INFO:teuthology.orchestra.run.smithi191.stdout: debug mgr = 20 2023-03-24T23:19:00.233 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-03-24T23:19:00.234 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-03-24T23:19:00.234 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-03-24T23:19:00.234 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-03-24T23:19:00.234 INFO:teuthology.orchestra.run.smithi191.stdout: mgr/telemetry/nag = false 2023-03-24T23:19:00.235 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.235 INFO:teuthology.orchestra.run.smithi191.stdout:[mon] 2023-03-24T23:19:00.235 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-03-24T23:19:00.235 INFO:teuthology.orchestra.run.smithi191.stdout: debug mon = 20 2023-03-24T23:19:00.236 INFO:teuthology.orchestra.run.smithi191.stdout: debug paxos = 20 2023-03-24T23:19:00.236 INFO:teuthology.orchestra.run.smithi191.stdout: debug auth = 20 2023-03-24T23:19:00.236 INFO:teuthology.orchestra.run.smithi191.stdout: mon data avail warn = 5 2023-03-24T23:19:00.236 INFO:teuthology.orchestra.run.smithi191.stdout: mon mgr mkfs grace = 240 2023-03-24T23:19:00.237 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min pgs per osd = 4 2023-03-24T23:19:00.237 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd reporter subtree level = osd 2023-03-24T23:19:00.237 INFO:teuthology.orchestra.run.smithi191.stdout: mon osd prime pg temp = true 2023-03-24T23:19:00.238 INFO:teuthology.orchestra.run.smithi191.stdout: mon reweight min bytes per osd = 10 2023-03-24T23:19:00.238 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.238 INFO:teuthology.orchestra.run.smithi191.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-24T23:19:00.238 INFO:teuthology.orchestra.run.smithi191.stdout: auth mon ticket ttl = 660 # 11m 2023-03-24T23:19:00.239 INFO:teuthology.orchestra.run.smithi191.stdout: auth service ticket ttl = 240 # 4m 2023-03-24T23:19:00.239 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.239 INFO:teuthology.orchestra.run.smithi191.stdout: # don't complain about insecure global_id in the test suite 2023-03-24T23:19:00.239 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-24T23:19:00.240 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-24T23:19:00.240 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.240 INFO:teuthology.orchestra.run.smithi191.stdout: # 1m isn't quite enough 2023-03-24T23:19:00.240 INFO:teuthology.orchestra.run.smithi191.stdout: mon_down_mkfs_grace = 2m 2023-03-24T23:19:00.241 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.241 INFO:teuthology.orchestra.run.smithi191.stdout: mon_warn_on_filestore_osds = false 2023-03-24T23:19:00.241 INFO:teuthology.orchestra.run.smithi191.stdout: mon op complaint time = 120 2023-03-24T23:19:00.241 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:00.242 INFO:teuthology.orchestra.run.smithi191.stdout:[client] 2023-03-24T23:19:00.242 INFO:teuthology.orchestra.run.smithi191.stdout: rgw cache enabled = true 2023-03-24T23:19:00.242 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable ops log = true 2023-03-24T23:19:00.243 INFO:teuthology.orchestra.run.smithi191.stdout: rgw enable usage log = true 2023-03-24T23:19:00.243 INFO:teuthology.orchestra.run.smithi191.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-24T23:19:00.243 INFO:teuthology.orchestra.run.smithi191.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-24T23:19:00.243 INFO:teuthology.orchestra.run.smithi191.stdout: client mount timeout = 600 2023-03-24T23:19:00.244 INFO:teuthology.orchestra.run.smithi191.stdout: debug client = 20 2023-03-24T23:19:00.244 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-03-24T23:19:00.244 INFO:teuthology.orchestra.run.smithi191.stdout: pid file = "" 2023-03-24T23:19:00.244 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2023-03-24T23:19:00.245 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2023-03-24T23:19:00.245 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.a] 2023-03-24T23:19:00.245 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.b] 2023-03-24T23:19:00.245 INFO:teuthology.orchestra.run.smithi191.stdout:[mon.c] 2023-03-24T23:19:00.246 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2023-03-24T23:19:00.246 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2023-03-24T23:19:00.246 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2023-03-24T23:19:00.246 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2023-03-24T23:19:00.247 INFO:teuthology.orchestra.run.smithi191.stdout:[mds] 2023-03-24T23:19:00.247 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds = 20 2023-03-24T23:19:00.247 INFO:teuthology.orchestra.run.smithi191.stdout: debug mds balancer = 20 2023-03-24T23:19:00.247 INFO:teuthology.orchestra.run.smithi191.stdout: debug ms = 1 2023-03-24T23:19:00.248 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug frag = True 2023-03-24T23:19:00.248 INFO:teuthology.orchestra.run.smithi191.stdout: mds debug scatterstat = True 2023-03-24T23:19:00.248 INFO:teuthology.orchestra.run.smithi191.stdout: mds op complaint time = 180 2023-03-24T23:19:00.249 INFO:teuthology.orchestra.run.smithi191.stdout: mds verify scatter = True 2023-03-24T23:19:00.249 INFO:teuthology.orchestra.run.smithi191.stdout: osd op complaint time = 180 2023-03-24T23:19:00.249 INFO:teuthology.orchestra.run.smithi191.stdout: rados mon op timeout = 900 2023-03-24T23:19:00.249 INFO:teuthology.orchestra.run.smithi191.stdout: rados osd op timeout = 900 2023-03-24T23:19:00.251 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2023-03-24T23:19:00.252 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2023-03-24T23:19:00.252 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-03-24T23:19:00.252 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2023-03-24T23:19:00.253 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.253 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2023-03-24T23:19:00.253 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.253 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2023-03-24T23:19:00.254 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.254 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2023-03-24T23:19:00.254 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2023-03-24T23:19:00.254 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.255 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2023-03-24T23:19:00.255 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2023-03-24T23:19:00.255 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.256 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2023-03-24T23:19:00.256 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2023-03-24T23:19:00.256 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.256 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2023-03-24T23:19:00.257 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2023-03-24T23:19:00.257 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.257 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2023-03-24T23:19:00.257 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.258 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2023-03-24T23:19:00.258 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2023-03-24T23:19:00.258 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2023-03-24T23:19:00.259 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2023-03-24T23:19:00.259 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2023-03-24T23:19:00.259 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2023-03-24T23:19:00.259 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2023-03-24T23:19:00.260 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-03-24T23:19:00.260 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2023-03-24T23:19:00.260 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2023-03-24T23:19:00.260 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.261 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-03-24T23:19:00.261 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.261 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2023-03-24T23:19:00.262 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.262 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2023-03-24T23:19:00.262 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.262 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2023-03-24T23:19:00.263 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2023-03-24T23:19:00.263 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2023-03-24T23:19:00.263 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = "172.21.15.191,172.21.15.196,[v2:172.21.15.196:3301,v1:172.21.15.196:6790]" 2023-03-24T23:19:00.263 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on skipped message = False 2023-03-24T23:19:00.264 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = 72b355de-1f4b-4f02-899f-839bc265bc0f 2023-03-24T23:19:00.264 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.264 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2023-03-24T23:19:00.265 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2023-03-24T23:19:00.265 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.265 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2023-03-24T23:19:00.265 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2023-03-24T23:19:00.266 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.266 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2023-03-24T23:19:00.266 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2023-03-24T23:19:00.266 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.267 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2023-03-24T23:19:00.267 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = false 2023-03-24T23:19:00.267 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2023-03-24T23:19:00.267 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.267 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2023-03-24T23:19:00.267 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2023-03-24T23:19:00.267 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.267 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2023-03-24T23:19:00.268 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.268 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2023-03-24T23:19:00.268 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.268 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2023-03-24T23:19:00.268 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.268 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2023-03-24T23:19:00.268 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2023-03-24T23:19:00.269 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.269 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2023-03-24T23:19:00.269 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2023-03-24T23:19:00.269 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2023-03-24T23:19:00.269 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2023-03-24T23:19:00.269 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 20 2023-03-24T23:19:00.269 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 20 2023-03-24T23:19:00.269 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-03-24T23:19:00.270 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2023-03-24T23:19:00.270 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 10 2023-03-24T23:19:00.270 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2023-03-24T23:19:00.270 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2023-03-24T23:19:00.270 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2023-03-24T23:19:00.270 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2023-03-24T23:19:00.270 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2023-03-24T23:19:00.271 INFO:teuthology.orchestra.run.smithi196.stdout: osd op complaint time = 180 2023-03-24T23:19:00.271 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.271 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2023-03-24T23:19:00.271 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-03-24T23:19:00.271 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2023-03-24T23:19:00.271 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2023-03-24T23:19:00.271 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2023-03-24T23:19:00.271 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2023-03-24T23:19:00.272 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2023-03-24T23:19:00.272 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2023-03-24T23:19:00.272 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.272 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2023-03-24T23:19:00.272 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-03-24T23:19:00.272 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2023-03-24T23:19:00.272 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2023-03-24T23:19:00.272 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2023-03-24T23:19:00.273 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2023-03-24T23:19:00.273 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2023-03-24T23:19:00.273 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2023-03-24T23:19:00.273 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2023-03-24T23:19:00.273 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2023-03-24T23:19:00.273 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2023-03-24T23:19:00.273 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.274 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-03-24T23:19:00.274 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2023-03-24T23:19:00.274 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2023-03-24T23:19:00.274 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.274 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2023-03-24T23:19:00.274 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-03-24T23:19:00.274 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-03-24T23:19:00.274 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.275 INFO:teuthology.orchestra.run.smithi196.stdout: # 1m isn't quite enough 2023-03-24T23:19:00.275 INFO:teuthology.orchestra.run.smithi196.stdout: mon_down_mkfs_grace = 2m 2023-03-24T23:19:00.275 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.275 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_filestore_osds = false 2023-03-24T23:19:00.275 INFO:teuthology.orchestra.run.smithi196.stdout: mon op complaint time = 120 2023-03-24T23:19:00.275 INFO:teuthology.orchestra.run.smithi196.stdout: 2023-03-24T23:19:00.275 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2023-03-24T23:19:00.275 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2023-03-24T23:19:00.276 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2023-03-24T23:19:00.276 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2023-03-24T23:19:00.276 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-03-24T23:19:00.276 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-03-24T23:19:00.276 INFO:teuthology.orchestra.run.smithi196.stdout: client mount timeout = 600 2023-03-24T23:19:00.276 INFO:teuthology.orchestra.run.smithi196.stdout: debug client = 20 2023-03-24T23:19:00.276 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-03-24T23:19:00.277 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = "" 2023-03-24T23:19:00.277 INFO:teuthology.orchestra.run.smithi196.stdout: rados mon op timeout = 900 2023-03-24T23:19:00.277 INFO:teuthology.orchestra.run.smithi196.stdout: rados osd op timeout = 900 2023-03-24T23:19:00.277 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2023-03-24T23:19:00.277 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.b] 2023-03-24T23:19:00.277 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.c] 2023-03-24T23:19:00.277 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.a] 2023-03-24T23:19:00.277 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.c] 2023-03-24T23:19:00.278 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.b] 2023-03-24T23:19:00.278 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.d] 2023-03-24T23:19:00.278 INFO:teuthology.orchestra.run.smithi196.stdout:[mds] 2023-03-24T23:19:00.278 INFO:teuthology.orchestra.run.smithi196.stdout: debug mds = 20 2023-03-24T23:19:00.278 INFO:teuthology.orchestra.run.smithi196.stdout: debug mds balancer = 20 2023-03-24T23:19:00.278 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2023-03-24T23:19:00.278 INFO:teuthology.orchestra.run.smithi196.stdout: mds debug frag = True 2023-03-24T23:19:00.278 INFO:teuthology.orchestra.run.smithi196.stdout: mds debug scatterstat = True 2023-03-24T23:19:00.279 INFO:teuthology.orchestra.run.smithi196.stdout: mds op complaint time = 180 2023-03-24T23:19:00.279 INFO:teuthology.orchestra.run.smithi196.stdout: mds verify scatter = True 2023-03-24T23:19:00.279 INFO:teuthology.orchestra.run.smithi196.stdout: osd op complaint time = 180 2023-03-24T23:19:00.279 INFO:teuthology.orchestra.run.smithi196.stdout: rados mon op timeout = 900 2023-03-24T23:19:00.279 INFO:teuthology.orchestra.run.smithi196.stdout: rados osd op timeout = 900 2023-03-24T23:19:00.279 INFO:tasks.ceph:Creating admin key on mon.a... 2023-03-24T23:19:00.280 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-03-24T23:19:00.348 INFO:tasks.ceph:Copying monmap to all nodes... 2023-03-24T23:19:00.349 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:00.349 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-03-24T23:19:00.365 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:00.365 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-03-24T23:19:00.421 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi191.front.sepia.ceph.com 2023-03-24T23:19:00.421 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:00.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-24T23:19:00.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-24T23:19:00.496 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:00.497 DEBUG:teuthology.orchestra.run.smithi191:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:00.553 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2023-03-24T23:19:00.553 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:00.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.keyring 2023-03-24T23:19:00.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-03-24T23:19:00.590 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:00.591 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:00.647 INFO:tasks.ceph:Setting up mon nodes... 2023-03-24T23:19:00.648 INFO:tasks.ceph:Setting up mgr nodes... 2023-03-24T23:19:00.648 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-03-24T23:19:00.701 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-03-24T23:19:00.703 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-03-24T23:19:00.766 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-03-24T23:19:00.769 INFO:tasks.ceph:Setting up mds nodes... 2023-03-24T23:19:00.769 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-03-24T23:19:00.822 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-03-24T23:19:00.824 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-03-24T23:19:00.850 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-03-24T23:19:00.943 INFO:teuthology.orchestra.run.smithi191.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-03-24T23:19:00.945 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-03-24T23:19:00.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-03-24T23:19:01.023 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-03-24T23:19:01.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-03-24T23:19:01.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-03-24T23:19:01.144 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-03-24T23:19:01.147 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-03-24T23:19:01.172 INFO:tasks.ceph_client:Setting up client nodes... 2023-03-24T23:19:01.172 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-03-24T23:19:01.218 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-03-24T23:19:01.230 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-03-24T23:19:01.273 INFO:teuthology.orchestra.run.smithi191.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-03-24T23:19:01.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-03-24T23:19:01.328 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-03-24T23:19:01.341 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-03-24T23:19:01.383 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-03-24T23:19:01.396 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-03-24T23:19:01.396 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi191.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-03-24T23:19:01.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-03-24T23:19:01.422 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-24T23:19:01.423 INFO:tasks.ceph:role: osd.0 2023-03-24T23:19:01.423 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com 2023-03-24T23:19:01.423 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-24T23:19:01.497 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-24T23:19:01.497 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-24T23:19:01.497 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-24T23:19:01.497 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-24T23:19:01.497 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-24T23:19:01.497 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-03-24T23:19:01.498 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-24T23:19:01.498 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-24T23:19:01.498 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-24T23:19:01.498 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-24T23:19:01.930 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-03-24T23:19:01.932 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-03-24T23:19:01.932 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-03-24T23:19:02.111 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-03-24T23:19:02.143 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-03-24T23:19:02.209 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-24T23:19:02.210 INFO:tasks.ceph:role: osd.1 2023-03-24T23:19:02.210 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com 2023-03-24T23:19:02.210 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-24T23:19:02.282 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-24T23:19:02.283 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-24T23:19:02.283 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-24T23:19:02.283 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-24T23:19:02.283 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-24T23:19:02.283 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-03-24T23:19:02.283 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-24T23:19:02.283 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-24T23:19:02.284 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-24T23:19:02.284 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-24T23:19:02.705 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-03-24T23:19:02.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-03-24T23:19:02.707 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-03-24T23:19:02.820 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-03-24T23:19:02.853 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-03-24T23:19:02.920 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-24T23:19:02.921 INFO:tasks.ceph:role: osd.2 2023-03-24T23:19:02.921 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com 2023-03-24T23:19:02.921 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-24T23:19:02.994 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-24T23:19:02.994 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-24T23:19:02.995 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-24T23:19:02.995 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-24T23:19:02.995 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-24T23:19:02.995 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-03-24T23:19:02.995 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-24T23:19:02.995 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-24T23:19:02.995 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-24T23:19:02.996 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-24T23:19:03.440 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-03-24T23:19:03.442 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-03-24T23:19:03.442 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-03-24T23:19:03.603 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-03-24T23:19:03.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-03-24T23:19:03.709 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-03-24T23:19:03.709 INFO:tasks.ceph:role: osd.3 2023-03-24T23:19:03.709 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com 2023-03-24T23:19:03.710 DEBUG:teuthology.orchestra.run.smithi191:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-24T23:19:03.784 INFO:teuthology.orchestra.run.smithi191.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-24T23:19:03.785 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-24T23:19:03.785 INFO:teuthology.orchestra.run.smithi191.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-24T23:19:03.785 INFO:teuthology.orchestra.run.smithi191.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-24T23:19:03.785 INFO:teuthology.orchestra.run.smithi191.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-24T23:19:03.785 INFO:teuthology.orchestra.run.smithi191.stdout: = sunit=0 swidth=0 blks 2023-03-24T23:19:03.785 INFO:teuthology.orchestra.run.smithi191.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-24T23:19:03.785 INFO:teuthology.orchestra.run.smithi191.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-24T23:19:03.786 INFO:teuthology.orchestra.run.smithi191.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-24T23:19:03.786 INFO:teuthology.orchestra.run.smithi191.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-24T23:19:04.212 INFO:teuthology.orchestra.run.smithi191.stdout:Discarding blocks...Done. 2023-03-24T23:19:04.214 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi191.front.sepia.ceph.com -o noatime 2023-03-24T23:19:04.214 DEBUG:teuthology.orchestra.run.smithi191:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-03-24T23:19:04.386 DEBUG:teuthology.orchestra.run.smithi191:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-03-24T23:19:04.422 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:04.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:04.527+0000 7f3036fa8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-03-24T23:19:04.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:04.527+0000 7f3036fa8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-03-24T23:19:04.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:04.527+0000 7f3036fa8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-24T23:19:04.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:04.527+0000 7f3036fa8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-24T23:19:04.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:04.527+0000 7f3036fa8640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-03-24T23:19:04.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:04.527+0000 7f3036fa8640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-03-24T23:19:07.395 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-03-24T23:19:07.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:07.518 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:07.517+0000 7fc4c1e29640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-03-24T23:19:07.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:07.517+0000 7fc4c1e29640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-03-24T23:19:07.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:07.517+0000 7fc4c1e29640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-24T23:19:07.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:07.517+0000 7fc4c1e29640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-24T23:19:07.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:07.517+0000 7fc4c1e29640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-03-24T23:19:07.519 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:07.518+0000 7fc4c1e29640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-03-24T23:19:10.396 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-03-24T23:19:10.426 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:10.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:10.513+0000 7f750149f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-03-24T23:19:10.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:10.514+0000 7f750149f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-03-24T23:19:10.515 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:10.514+0000 7f750149f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-24T23:19:10.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:10.514+0000 7f750149f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-24T23:19:10.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:10.514+0000 7f750149f640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-03-24T23:19:10.516 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:10.514+0000 7f750149f640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-03-24T23:19:13.383 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-03-24T23:19:13.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:13.505 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:13.504+0000 7f956eaec640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-03-24T23:19:13.506 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:13.504+0000 7f956eaec640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-03-24T23:19:13.506 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:13.504+0000 7f956eaec640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-24T23:19:13.506 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:13.504+0000 7f956eaec640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-24T23:19:13.506 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:13.504+0000 7f956eaec640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-03-24T23:19:13.507 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:13.505+0000 7f956eaec640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-03-24T23:19:16.373 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-03-24T23:19:16.405 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-03-24T23:19:16.438 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-24T23:19:16.438 INFO:tasks.ceph:role: osd.4 2023-03-24T23:19:16.439 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2023-03-24T23:19:16.439 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-03-24T23:19:16.516 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-03-24T23:19:16.517 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-24T23:19:16.517 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-24T23:19:16.517 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-24T23:19:16.517 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-24T23:19:16.517 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2023-03-24T23:19:16.517 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-24T23:19:16.518 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-24T23:19:16.518 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-24T23:19:16.518 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-24T23:19:16.952 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2023-03-24T23:19:16.954 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2023-03-24T23:19:16.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-03-24T23:19:17.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-03-24T23:19:17.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-03-24T23:19:17.209 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-24T23:19:17.209 INFO:tasks.ceph:role: osd.5 2023-03-24T23:19:17.210 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2023-03-24T23:19:17.210 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-03-24T23:19:17.286 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-03-24T23:19:17.286 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-24T23:19:17.286 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-24T23:19:17.286 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-24T23:19:17.287 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-24T23:19:17.287 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2023-03-24T23:19:17.287 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-24T23:19:17.287 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-24T23:19:17.287 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-24T23:19:17.287 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-24T23:19:17.705 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2023-03-24T23:19:17.707 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2023-03-24T23:19:17.707 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-03-24T23:19:17.830 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-03-24T23:19:17.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-03-24T23:19:17.932 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-24T23:19:17.933 INFO:tasks.ceph:role: osd.6 2023-03-24T23:19:17.933 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2023-03-24T23:19:17.933 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-03-24T23:19:18.008 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-03-24T23:19:18.009 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-24T23:19:18.009 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-24T23:19:18.009 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-24T23:19:18.009 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-24T23:19:18.009 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2023-03-24T23:19:18.009 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-24T23:19:18.009 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-24T23:19:18.010 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-24T23:19:18.010 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-24T23:19:18.457 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2023-03-24T23:19:18.459 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2023-03-24T23:19:18.459 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-03-24T23:19:18.626 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-03-24T23:19:18.661 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-03-24T23:19:18.731 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-03-24T23:19:18.731 INFO:tasks.ceph:role: osd.7 2023-03-24T23:19:18.732 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com 2023-03-24T23:19:18.732 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-03-24T23:19:18.808 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-03-24T23:19:18.808 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2023-03-24T23:19:18.808 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-03-24T23:19:18.809 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-03-24T23:19:18.809 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-03-24T23:19:18.809 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2023-03-24T23:19:18.809 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-03-24T23:19:18.809 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-03-24T23:19:18.809 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-03-24T23:19:18.809 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-03-24T23:19:19.249 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2023-03-24T23:19:19.250 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2023-03-24T23:19:19.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-03-24T23:19:19.423 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-03-24T23:19:19.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:19.589 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:19.588+0000 7f2108e24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-03-24T23:19:19.590 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:19.589+0000 7f2108e24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-03-24T23:19:19.590 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:19.589+0000 7f2108e24640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-24T23:19:19.590 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:19.589+0000 7f2108e24640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-24T23:19:19.590 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:19.589+0000 7f2108e24640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-03-24T23:19:19.591 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:19.590+0000 7f2108e24640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-03-24T23:19:22.461 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-03-24T23:19:22.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:22.583 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:22.581+0000 7f905e329640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-03-24T23:19:22.583 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:22.582+0000 7f905e329640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-03-24T23:19:22.583 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:22.582+0000 7f905e329640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-24T23:19:22.584 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:22.582+0000 7f905e329640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-24T23:19:22.584 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:22.582+0000 7f905e329640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-03-24T23:19:22.584 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:22.582+0000 7f905e329640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-03-24T23:19:25.452 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-03-24T23:19:25.483 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:25.577 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:25.575+0000 7f6e6b170640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-03-24T23:19:25.577 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:25.576+0000 7f6e6b170640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-03-24T23:19:25.578 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:25.576+0000 7f6e6b170640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-24T23:19:25.578 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:25.576+0000 7f6e6b170640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-24T23:19:25.578 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:25.576+0000 7f6e6b170640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-03-24T23:19:25.578 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:25.576+0000 7f6e6b170640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-03-24T23:19:28.188 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-24T23:19:28.192 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-24T23:19:28.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-03-24T23:19:28.479 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:28.570 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:28.569+0000 7fa3fb663640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-03-24T23:19:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:28.569+0000 7fa3fb663640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-03-24T23:19:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:28.569+0000 7fa3fb663640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-24T23:19:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:28.569+0000 7fa3fb663640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-24T23:19:28.571 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:28.569+0000 7fa3fb663640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-03-24T23:19:28.572 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:28.570+0000 7fa3fb663640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-03-24T23:19:31.430 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-03-24T23:19:31.461 INFO:tasks.ceph:Reading keys from all nodes... 2023-03-24T23:19:31.461 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:31.461 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-03-24T23:19:31.494 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:31.494 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-03-24T23:19:31.563 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:31.563 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-03-24T23:19:31.631 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:31.632 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-03-24T23:19:31.699 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:31.699 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-03-24T23:19:31.766 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:31.767 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-03-24T23:19:31.834 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:31.834 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-03-24T23:19:31.900 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:31.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-03-24T23:19:31.931 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:31.932 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-03-24T23:19:32.001 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:32.001 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-03-24T23:19:32.069 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:32.070 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-03-24T23:19:32.138 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:32.138 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-03-24T23:19:32.206 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:32.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-03-24T23:19:32.273 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:32.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-03-24T23:19:32.340 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:32.340 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-03-24T23:19:32.357 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:32.358 DEBUG:teuthology.orchestra.run.smithi191:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-03-24T23:19:32.415 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:32.415 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-03-24T23:19:32.432 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:32.433 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-03-24T23:19:32.489 INFO:tasks.ceph:Adding keys to all mons... 2023-03-24T23:19:32.490 DEBUG:teuthology.orchestra.run.smithi191:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-24T23:19:32.492 DEBUG:teuthology.orchestra.run.smithi196:> sudo tee -a /etc/ceph/ceph.keyring 2023-03-24T23:19:32.533 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.y] 2023-03-24T23:19:32.533 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDkLx5kglbZKRAA9CM8yjz0t3tMcvQ+KH0clg== 2023-03-24T23:19:32.533 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.a] 2023-03-24T23:19:32.533 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDkLx5k2uYJMRAAl4mM+8Xjlf+7ULLn8KHTVg== 2023-03-24T23:19:32.533 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.c] 2023-03-24T23:19:32.533 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDkLx5ksvo9OBAAo0nhJrl+uzBUnh3MBJwykw== 2023-03-24T23:19:32.534 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.0] 2023-03-24T23:19:32.534 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDoLx5kxDCEHxAA1CZ7SqXy74aTdP/wgsyX0g== 2023-03-24T23:19:32.534 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.1] 2023-03-24T23:19:32.534 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDrLx5kgubrHhAAAeWPAv8Bsgy1VdpwZKVqbg== 2023-03-24T23:19:32.534 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.2] 2023-03-24T23:19:32.534 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDuLx5k8Yu6HhAAiKJ+bpb/lsG+AHqndka2DQ== 2023-03-24T23:19:32.534 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.3] 2023-03-24T23:19:32.535 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDxLx5kV1YsHhAADD0dtze1ZaMwZRzTXtMfzA== 2023-03-24T23:19:32.535 INFO:teuthology.orchestra.run.smithi191.stdout:[mgr.x] 2023-03-24T23:19:32.535 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDkLx5kX8m6LRAA4JYC+c0LHafv3PDNQ8xokA== 2023-03-24T23:19:32.535 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.b] 2023-03-24T23:19:32.535 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDlLx5kAEF2ARAAW2lbkGgkamHeTyKQQg4oew== 2023-03-24T23:19:32.535 INFO:teuthology.orchestra.run.smithi191.stdout:[mds.d] 2023-03-24T23:19:32.535 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDlLx5k63yoCBAAB8mVq5g67oP7JY61ZiNiiQ== 2023-03-24T23:19:32.535 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.4] 2023-03-24T23:19:32.536 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQD3Lx5kbKghIxAAJzdCr++4LNf5IZ5L+s9RDA== 2023-03-24T23:19:32.536 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.5] 2023-03-24T23:19:32.536 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQD6Lx5kR03EIhAAJlqJvbcStDGwghhFAU9SYw== 2023-03-24T23:19:32.536 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.6] 2023-03-24T23:19:32.536 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQD9Lx5kbfxnIhAA83V08UTC1a+/gkFFc/gjLQ== 2023-03-24T23:19:32.536 INFO:teuthology.orchestra.run.smithi191.stdout:[osd.7] 2023-03-24T23:19:32.536 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQAAMB5k5S0IIhAAsbwHKONzxpTISi2hpfX4/Q== 2023-03-24T23:19:32.537 INFO:teuthology.orchestra.run.smithi191.stdout:[client.0] 2023-03-24T23:19:32.537 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDlLx5kw84HDRAA9Q+SZGf7JuADAUhHu3rcwg== 2023-03-24T23:19:32.537 INFO:teuthology.orchestra.run.smithi191.stdout:[client.2] 2023-03-24T23:19:32.537 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDlLx5ko2pOEBAA6r0tNilbpsJnvKQ5qG3JTw== 2023-03-24T23:19:32.537 INFO:teuthology.orchestra.run.smithi191.stdout:[client.1] 2023-03-24T23:19:32.537 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDlLx5kLbmiExAAc2V/esCpxKSwcQv0t2AGHw== 2023-03-24T23:19:32.537 INFO:teuthology.orchestra.run.smithi191.stdout:[client.3] 2023-03-24T23:19:32.538 INFO:teuthology.orchestra.run.smithi191.stdout: key = AQDlLx5kE/PmFhAAddr3vgMlvap0PzCOUFPo6g== 2023-03-24T23:19:32.555 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.y] 2023-03-24T23:19:32.556 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDkLx5kglbZKRAA9CM8yjz0t3tMcvQ+KH0clg== 2023-03-24T23:19:32.556 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.a] 2023-03-24T23:19:32.556 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDkLx5k2uYJMRAAl4mM+8Xjlf+7ULLn8KHTVg== 2023-03-24T23:19:32.556 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.c] 2023-03-24T23:19:32.556 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDkLx5ksvo9OBAAo0nhJrl+uzBUnh3MBJwykw== 2023-03-24T23:19:32.556 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2023-03-24T23:19:32.556 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDoLx5kxDCEHxAA1CZ7SqXy74aTdP/wgsyX0g== 2023-03-24T23:19:32.557 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2023-03-24T23:19:32.557 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDrLx5kgubrHhAAAeWPAv8Bsgy1VdpwZKVqbg== 2023-03-24T23:19:32.557 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2023-03-24T23:19:32.557 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDuLx5k8Yu6HhAAiKJ+bpb/lsG+AHqndka2DQ== 2023-03-24T23:19:32.557 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.3] 2023-03-24T23:19:32.557 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDxLx5kV1YsHhAADD0dtze1ZaMwZRzTXtMfzA== 2023-03-24T23:19:32.557 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.x] 2023-03-24T23:19:32.558 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDkLx5kX8m6LRAA4JYC+c0LHafv3PDNQ8xokA== 2023-03-24T23:19:32.558 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.b] 2023-03-24T23:19:32.558 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDlLx5kAEF2ARAAW2lbkGgkamHeTyKQQg4oew== 2023-03-24T23:19:32.558 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.d] 2023-03-24T23:19:32.558 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDlLx5k63yoCBAAB8mVq5g67oP7JY61ZiNiiQ== 2023-03-24T23:19:32.558 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.4] 2023-03-24T23:19:32.558 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQD3Lx5kbKghIxAAJzdCr++4LNf5IZ5L+s9RDA== 2023-03-24T23:19:32.558 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.5] 2023-03-24T23:19:32.559 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQD6Lx5kR03EIhAAJlqJvbcStDGwghhFAU9SYw== 2023-03-24T23:19:32.559 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.6] 2023-03-24T23:19:32.559 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQD9Lx5kbfxnIhAA83V08UTC1a+/gkFFc/gjLQ== 2023-03-24T23:19:32.559 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.7] 2023-03-24T23:19:32.559 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQAAMB5k5S0IIhAAsbwHKONzxpTISi2hpfX4/Q== 2023-03-24T23:19:32.559 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2023-03-24T23:19:32.559 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDlLx5kw84HDRAA9Q+SZGf7JuADAUhHu3rcwg== 2023-03-24T23:19:32.560 INFO:teuthology.orchestra.run.smithi196.stdout:[client.2] 2023-03-24T23:19:32.560 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDlLx5ko2pOEBAA6r0tNilbpsJnvKQ5qG3JTw== 2023-03-24T23:19:32.560 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2023-03-24T23:19:32.560 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDlLx5kLbmiExAAc2V/esCpxKSwcQv0t2AGHw== 2023-03-24T23:19:32.560 INFO:teuthology.orchestra.run.smithi196.stdout:[client.3] 2023-03-24T23:19:32.560 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDlLx5kE/PmFhAAddr3vgMlvap0PzCOUFPo6g== 2023-03-24T23:19:32.561 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-24T23:19:32.581 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-24T23:19:32.656 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-24T23:19:32.658 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-24T23:19:32.715 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-24T23:19:32.717 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-24T23:19:32.773 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:32.775 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:32.827 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:32.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:32.881 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:32.883 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:32.936 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:32.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:32.990 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-24T23:19:32.992 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-03-24T23:19:33.045 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-24T23:19:33.047 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-24T23:19:33.103 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-24T23:19:33.105 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-03-24T23:19:33.160 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:33.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:33.215 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:33.218 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:33.271 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:33.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:33.326 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:33.328 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-03-24T23:19:33.381 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-24T23:19:33.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-24T23:19:33.438 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-24T23:19:33.440 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-24T23:19:33.496 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-24T23:19:33.498 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-24T23:19:33.554 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-24T23:19:33.556 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-03-24T23:19:33.612 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-03-24T23:19:33.613 DEBUG:teuthology.orchestra.run.smithi191:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-03-24T23:19:33.639 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-24T23:19:33.736 DEBUG:teuthology.orchestra.run.smithi191:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-03-24T23:19:33.763 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-03-24T23:19:33.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-24T23:19:33.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-03-24T23:19:33.915 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-03-24T23:19:33.981 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-03-24T23:19:34.078 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-03-24T23:19:34.104 DEBUG:teuthology.orchestra.run.smithi191:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:34.106 DEBUG:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-03-24T23:19:34.160 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-03-24T23:19:34.160 INFO:tasks.ceph.mon.a:Restarting daemon 2023-03-24T23:19:34.161 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-03-24T23:19:34.163 INFO:tasks.ceph.mon.a:Started 2023-03-24T23:19:34.163 INFO:tasks.ceph.mon.b:Restarting daemon 2023-03-24T23:19:34.163 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-03-24T23:19:34.202 INFO:tasks.ceph.mon.b:Started 2023-03-24T23:19:34.202 INFO:tasks.ceph.mon.c:Restarting daemon 2023-03-24T23:19:34.202 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-03-24T23:19:34.204 INFO:tasks.ceph.mon.c:Started 2023-03-24T23:19:34.204 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-03-24T23:19:34.205 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-03-24T23:19:34.205 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-03-24T23:19:34.207 INFO:tasks.ceph.mgr.y:Started 2023-03-24T23:19:34.207 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-03-24T23:19:34.207 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-03-24T23:19:34.209 INFO:tasks.ceph.mgr.x:Started 2023-03-24T23:19:34.209 INFO:tasks.ceph:Setting crush tunables to default 2023-03-24T23:19:34.210 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd crush tunables default 2023-03-24T23:19:34.332 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.331+0000 7f8685386700 1 Processor -- start 2023-03-24T23:19:34.332 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.331+0000 7f8685386700 1 -- start start 2023-03-24T23:19:34.333 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f8685386700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 0x7f86800c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:34.333 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f8685386700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86800b4050 0x7f86800b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:34.333 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f8685386700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86800c7440 con 0x7f868015a9f0 2023-03-24T23:19:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f8685386700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86800b4a30 con 0x7f86800b4050 2023-03-24T23:19:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f8685386700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86800ca250 con 0x7f868015a620 2023-03-24T23:19:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867e7fc700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f868015a9f0 0x7f868015e3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57344/0 (socket says 172.21.15.191:57344) 2023-03-24T23:19:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867f7fe700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f86800b4050 0x7f86800b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867e7fc700 1 -- 172.21.15.191:0/2262151575 learned_addr learned my addr 172.21.15.191:0/2262151575 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867effd700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 0x7f86800c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:34.334 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867f7fe700 1 -- 172.21.15.191:0/2262151575 >> v2:172.21.15.191:3300/0 conn(0x7f86800b4050 msgr2=0x7f86800b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-03-24T23:19:34.335 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867f7fe700 1 -- 172.21.15.191:0/2262151575 >> v2:172.21.15.191:3300/0 conn(0x7f86800b4050 msgr2=0x7f86800b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-24T23:19:34.335 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867effd700 1 -- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 msgr2=0x7f86800c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-03-24T23:19:34.335 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867f7fe700 1 --2- 172.21.15.191:0/2262151575 >> v2:172.21.15.191:3300/0 conn(0x7f86800b4050 0x7f86800b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-24T23:19:34.335 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867effd700 1 -- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 msgr2=0x7f86800c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-03-24T23:19:34.335 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867effd700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 0x7f86800c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-03-24T23:19:34.335 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867effd700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 0x7f86800c6f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-24T23:19:34.335 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.332+0000 7f867f7fe700 1 --2- 172.21.15.191:0/2262151575 >> v2:172.21.15.191:3300/0 conn(0x7f86800b4050 0x7f86800b44f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-03-24T23:19:34.524 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.522+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273143952 0 0) 0x7f86800c7440 con 0x7f868015a9f0 2023-03-24T23:19:34.525 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.522+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86680035b0 con 0x7f868015a9f0 2023-03-24T23:19:34.525 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.522+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 <== mon.0 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401846861 0 0) 0x7f86680035b0 con 0x7f868015a9f0 2023-03-24T23:19:34.525 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.522+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 msgr2=0x7f86800c6f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-24T23:19:34.525 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.522+0000 7f867dffb700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 0x7f86800c6f00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.525 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 >> v2:172.21.15.191:3300/0 conn(0x7f86800b4050 msgr2=0x7f86800b44f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-03-24T23:19:34.525 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 --2- 172.21.15.191:0/2262151575 >> v2:172.21.15.191:3300/0 conn(0x7f86800b4050 0x7f86800b44f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.525 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f868015ef10 con 0x7f868015a9f0 2023-03-24T23:19:34.526 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 <== mon.0 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1537907258 0 0) 0x7f8674001670 con 0x7f868015a9f0 2023-03-24T23:19:34.526 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 >> v1:172.21.15.191:6789/0 conn(0x7f868015a9f0 legacy=0x7f868015e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:34.526 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 0x7f8668003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:34.526 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f8668004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:34.526 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8668004d90 0x7f8668005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:34.526 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86800ca250 con 0x7f8668004030 2023-03-24T23:19:34.526 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866800bb20 con 0x7f8668004d90 2023-03-24T23:19:34.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f866800bce0 con 0x7f868015a620 2023-03-24T23:19:34.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867effd700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 0x7f8668003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:34.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867e7fc700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f8668004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:34.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867f7fe700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8668004d90 0x7f8668005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:34.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867e7fc700 1 -- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 msgr2=0x7f8668003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:34.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867e7fc700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f868015a620 0x7f8668003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867e7fc700 1 -- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8668004d90 msgr2=0x7f8668005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:34.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867e7fc700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8668004d90 0x7f8668005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867e7fc700 1 -- 172.21.15.191:0/2262151575 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86680035b0 con 0x7f8668004030 2023-03-24T23:19:34.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867e7fc700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f8668004850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8674001050 tx=0x7f8668005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a228b71bb58016a server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:34.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674007c60 con 0x7f8668004030 2023-03-24T23:19:34.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f867400ce80 con 0x7f8668004030 2023-03-24T23:19:34.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.523+0000 7f867dffb700 1 -- 172.21.15.191:0/2262151575 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674005ba0 con 0x7f8668004030 2023-03-24T23:19:34.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.524+0000 7f8685386700 1 -- 172.21.15.191:0/2262151575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 msgr2=0x7f8668004850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:34.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.524+0000 7f8685386700 1 --2- 172.21.15.191:0/2262151575 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f8668004850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8674001050 tx=0x7f8668005880 comp rx=0 tx=0).stop 2023-03-24T23:19:34.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.524+0000 7f867f7fe700 1 -- 172.21.15.191:0/2262151575 reap_dead start 2023-03-24T23:19:34.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.524+0000 7f8685386700 1 -- 172.21.15.191:0/2262151575 shutdown_connections 2023-03-24T23:19:34.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.524+0000 7f8685386700 1 -- 172.21.15.191:0/2262151575 >> 172.21.15.191:0/2262151575 conn(0x7f86800d0b50 msgr2=0x7f86800cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:34.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.524+0000 7f8685386700 1 -- 172.21.15.191:0/2262151575 shutdown_connections 2023-03-24T23:19:34.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.524+0000 7f8685386700 1 -- 172.21.15.191:0/2262151575 wait complete. 2023-03-24T23:19:34.531 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.526+0000 7f8685386700 1 Processor -- start 2023-03-24T23:19:34.531 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.526+0000 7f8685386700 1 -- start start 2023-03-24T23:19:34.531 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f8685386700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f86800c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:34.531 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f8685386700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86800c7910 0x7f868014c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:34.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f8685386700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8680153ee0 0x7f8680154330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:34.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f8685386700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8680156060 con 0x7f8668004030 2023-03-24T23:19:34.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f86800c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:34.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f86800c9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:49968/0 (socket says 172.21.15.191:49968) 2023-03-24T23:19:34.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 -- 172.21.15.191:0/366502947 learned_addr learned my addr 172.21.15.191:0/366502947 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:34.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8680154870 con 0x7f86800c7910 2023-03-24T23:19:34.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86801561f0 con 0x7f8680153ee0 2023-03-24T23:19:34.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867e7fc700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86800c7910 0x7f868014c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:34.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867f7fe700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8680153ee0 0x7f8680154330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:34.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 -- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8680153ee0 msgr2=0x7f8680154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:34.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8680153ee0 0x7f8680154330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 -- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86800c7910 msgr2=0x7f868014c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:34.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86800c7910 0x7f868014c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 -- 172.21.15.191:0/366502947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86800ca480 con 0x7f8668004030 2023-03-24T23:19:34.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867e7fc700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86800c7910 0x7f868014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:34.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.527+0000 7f867effd700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f86800c9a30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f867800afc0 tx=0x7f867800f430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:34.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.528+0000 7f86677fe700 1 -- 172.21.15.191:0/366502947 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678018070 con 0x7f8668004030 2023-03-24T23:19:34.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.528+0000 7f86677fe700 1 -- 172.21.15.191:0/366502947 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8678009ca0 con 0x7f8668004030 2023-03-24T23:19:34.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.528+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f868015f1f0 con 0x7f8668004030 2023-03-24T23:19:34.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.528+0000 7f86677fe700 1 -- 172.21.15.191:0/366502947 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8678003c60 con 0x7f8668004030 2023-03-24T23:19:34.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.528+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86800ca760 con 0x7f8668004030 2023-03-24T23:19:34.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.529+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86801564d0 con 0x7f8668004030 2023-03-24T23:19:34.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.529+0000 7f86677fe700 1 -- 172.21.15.191:0/366502947 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f867800fa50 con 0x7f8668004030 2023-03-24T23:19:34.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.529+0000 7f86677fe700 1 -- 172.21.15.191:0/366502947 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8678004700 con 0x7f8668004030 2023-03-24T23:19:34.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.532+0000 7f86677fe700 1 -- 172.21.15.191:0/366502947 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f8678003dc0 con 0x7f8668004030 2023-03-24T23:19:34.607 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.607+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f86800cb5c0 con 0x7f8668004030 2023-03-24T23:19:34.609 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.609+0000 7f86677fe700 1 -- 172.21.15.191:0/366502947 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f867800fd00 con 0x7f8668004030 2023-03-24T23:19:34.611 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 msgr2=0x7f86800c9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:34.611 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f86800c9a30 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f867800afc0 tx=0x7f867800f430 comp rx=0 tx=0).stop 2023-03-24T23:19:34.611 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 shutdown_connections 2023-03-24T23:19:34.611 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8668004030 0x7f86800c9a30 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.611 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8680153ee0 0x7f8680154330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.611 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 --2- 172.21.15.191:0/366502947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f86800c7910 0x7f868014c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:34.611 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 >> 172.21.15.191:0/366502947 conn(0x7f86800d0b50 msgr2=0x7f86800b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:34.612 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 shutdown_connections 2023-03-24T23:19:34.612 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:34.610+0000 7f8685386700 1 -- 172.21.15.191:0/366502947 wait complete. 2023-03-24T23:19:34.612 INFO:teuthology.orchestra.run.smithi191.stderr:adjusted tunables profile to default 2023-03-24T23:19:34.623 INFO:tasks.ceph:check_enable_crimson: False 2023-03-24T23:19:34.623 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-03-24T23:19:34.623 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:34.624 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-03-24T23:19:34.638 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:34.637+0000 7fc54dc46200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-24T23:19:34.642 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:34.642+0000 7f6bb5d9b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-03-24T23:19:34.649 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:34.650 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-03-24T23:19:34.703 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:34.702+0000 7fc54dc46200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-24T23:19:34.713 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:34.713+0000 7f6bb5d9b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-03-24T23:19:34.716 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:34.717 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-03-24T23:19:34.777 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:34.776+0000 7fc54dc46200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-24T23:19:34.785 DEBUG:teuthology.orchestra.run.smithi191:> set -ex 2023-03-24T23:19:34.785 DEBUG:teuthology.orchestra.run.smithi191:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-03-24T23:19:34.793 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:34.793+0000 7f6bb5d9b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-03-24T23:19:34.820 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:34.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-03-24T23:19:34.847 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:34.847 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-03-24T23:19:34.859 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:34.858+0000 7fc54dc46200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-24T23:19:34.878 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:34.878+0000 7f6bb5d9b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-03-24T23:19:34.884 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:34.884 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-03-24T23:19:34.950 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2023-03-24T23:19:34.950 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-03-24T23:19:35.009 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:35.008+0000 7fc54dc46200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-24T23:19:35.015 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 0ca3720a-fe6e-4b70-a32f-fef43c5da336 0 2023-03-24T23:19:35.020 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:35.020+0000 7f6bb5d9b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-03-24T23:19:35.176 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.175+0000 7f0beb182700 1 Processor -- start 2023-03-24T23:19:35.177 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.175+0000 7f0beb182700 1 -- start start 2023-03-24T23:19:35.178 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0beb182700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be40b23f0 0x7f0be406c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.179 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0beb182700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0be406acb0 0x7f0be40d1730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.179 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0beb182700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0be40cedc0 con 0x7f0be40ce2b0 2023-03-24T23:19:35.179 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0beb182700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0be40d1310 con 0x7f0be406acb0 2023-03-24T23:19:35.179 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0beb182700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be40ce800 con 0x7f0be40b23f0 2023-03-24T23:19:35.179 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0be8f1e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be40b23f0 0x7f0be406c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.179 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0be971f700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0be406acb0 0x7f0be40d1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.180 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0be8f1e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be40b23f0 0x7f0be406c690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:50178/0 (socket says 172.21.15.196:50178) 2023-03-24T23:19:35.180 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0be971f700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0be406acb0 0x7f0be40d1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35630/0 (socket says 172.21.15.196:35630) 2023-03-24T23:19:35.180 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.176+0000 7f0be8f1e700 1 -- 172.21.15.196:0/1206145480 learned_addr learned my addr 172.21.15.196:0/1206145480 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:35.180 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0be8f1e700 1 -- 172.21.15.196:0/1206145480 >> v2:172.21.15.196:3300/0 conn(0x7f0be406acb0 msgr2=0x7f0be40d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:35.180 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0be8f1e700 1 --2- 172.21.15.196:0/1206145480 >> v2:172.21.15.196:3300/0 conn(0x7f0be406acb0 0x7f0be40d1730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.180 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0be8f1e700 1 -- 172.21.15.196:0/1206145480 >> v1:172.21.15.196:6789/0 conn(0x7f0be40ce2b0 legacy=0x7f0be406a5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.180 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0be8f1e700 1 -- 172.21.15.196:0/1206145480 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be40ceae0 con 0x7f0be40b23f0 2023-03-24T23:19:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0be8f1e700 1 --2- 172.21.15.196:0/1206145480 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be40b23f0 0x7f0be406c690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098f0 tx=0x7f0bd8009c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94cd95f0b72a0ed8 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0be37fe700 1 -- 172.21.15.196:0/1206145480 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8011830 con 0x7f0be40b23f0 2023-03-24T23:19:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0be37fe700 1 -- 172.21.15.196:0/1206145480 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd8011990 con 0x7f0be40b23f0 2023-03-24T23:19:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0be37fe700 1 -- 172.21.15.196:0/1206145480 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd8011d20 con 0x7f0be40b23f0 2023-03-24T23:19:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0beb182700 1 -- 172.21.15.196:0/1206145480 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be40b23f0 msgr2=0x7f0be406c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.177+0000 7f0beb182700 1 --2- 172.21.15.196:0/1206145480 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be40b23f0 0x7f0be406c690 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0bd80098f0 tx=0x7f0bd8009c00 comp rx=0 tx=0).stop 2023-03-24T23:19:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.178+0000 7f0beb182700 1 -- 172.21.15.196:0/1206145480 shutdown_connections 2023-03-24T23:19:35.181 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.178+0000 7f0beb182700 1 --2- 172.21.15.196:0/1206145480 >> v2:172.21.15.196:3300/0 conn(0x7f0be406acb0 0x7f0be40d1730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.182 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.178+0000 7f0beb182700 1 --2- 172.21.15.196:0/1206145480 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be40b23f0 0x7f0be406c690 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.182 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.178+0000 7f0beb182700 1 -- 172.21.15.196:0/1206145480 >> 172.21.15.196:0/1206145480 conn(0x7f0be40d0920 msgr2=0x7f0be40d3250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:35.182 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.178+0000 7f0beb182700 1 -- 172.21.15.196:0/1206145480 shutdown_connections 2023-03-24T23:19:35.182 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.178+0000 7f0beb182700 1 -- 172.21.15.196:0/1206145480 wait complete. 2023-03-24T23:19:35.184 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.179+0000 7f0beb182700 1 Processor -- start 2023-03-24T23:19:35.184 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.180+0000 7f0beb182700 1 -- start start 2023-03-24T23:19:35.184 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0beb182700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be406acb0 0x7f0be41515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.184 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0beb182700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0be40b23f0 0x7f0be40af290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.184 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0beb182700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0be40ce2b0 0x7f0be40af7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.185 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0beb182700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4154ad0 con 0x7f0be40ce2b0 2023-03-24T23:19:35.185 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0beb182700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be41521c0 con 0x7f0be40b23f0 2023-03-24T23:19:35.185 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0beb182700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4152460 con 0x7f0be406acb0 2023-03-24T23:19:35.185 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be8f1e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be406acb0 0x7f0be41515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.185 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be3fff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0be40b23f0 0x7f0be40af290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.185 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be8f1e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be406acb0 0x7f0be41515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:50184/0 (socket says 172.21.15.196:50184) 2023-03-24T23:19:35.185 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be8f1e700 1 -- 172.21.15.196:0/2735631496 learned_addr learned my addr 172.21.15.196:0/2735631496 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be971f700 1 --2- 172.21.15.196:0/2735631496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0be40ce2b0 0x7f0be40af7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be3fff700 1 -- 172.21.15.196:0/2735631496 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be406acb0 msgr2=0x7f0be41515a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be3fff700 1 --2- 172.21.15.196:0/2735631496 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be406acb0 0x7f0be41515a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be3fff700 1 -- 172.21.15.196:0/2735631496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0be40ce2b0 msgr2=0x7f0be40af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be3fff700 1 --2- 172.21.15.196:0/2735631496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0be40ce2b0 0x7f0be40af7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.181+0000 7f0be3fff700 1 -- 172.21.15.196:0/2735631496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd8009550 con 0x7f0be40b23f0 2023-03-24T23:19:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.182+0000 7f0be3fff700 1 --2- 172.21.15.196:0/2735631496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0be40b23f0 0x7f0be40af290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0000f60 tx=0x7f0bd000c730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:35.186 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.182+0000 7f0be17fa700 1 -- 172.21.15.196:0/2735631496 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd00052d0 con 0x7f0be40b23f0 2023-03-24T23:19:35.187 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.182+0000 7f0be17fa700 1 -- 172.21.15.196:0/2735631496 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd0005430 con 0x7f0be40b23f0 2023-03-24T23:19:35.187 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.182+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be40b6080 con 0x7f0be40b23f0 2023-03-24T23:19:35.187 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.182+0000 7f0be17fa700 1 -- 172.21.15.196:0/2735631496 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bd00038c0 con 0x7f0be40b23f0 2023-03-24T23:19:35.187 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.182+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be40d0f30 con 0x7f0be40b23f0 2023-03-24T23:19:35.187 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.182+0000 7f0be17fa700 1 -- 172.21.15.196:0/2735631496 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f0bd0003d20 con 0x7f0be40b23f0 2023-03-24T23:19:35.187 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.182+0000 7f0be17fa700 1 -- 172.21.15.196:0/2735631496 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7f0bd001c780 con 0x7f0be40b23f0 2023-03-24T23:19:35.187 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.183+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be40b27c0 con 0x7f0be40b23f0 2023-03-24T23:19:35.188 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.185+0000 7f0be17fa700 1 -- 172.21.15.196:0/2735631496 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f0bd0017070 con 0x7f0be40b23f0 2023-03-24T23:19:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.280+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0ca3720a-fe6e-4b70-a32f-fef43c5da336", "id": 0} v 0) v1 -- 0x7f0be4074580 con 0x7f0be40b23f0 2023-03-24T23:19:35.286 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.285+0000 7f0be17fa700 1 -- 172.21.15.196:0/2735631496 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0ca3720a-fe6e-4b70-a32f-fef43c5da336", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0bd0011df0 con 0x7f0be40b23f0 2023-03-24T23:19:35.286 INFO:teuthology.orchestra.run.smithi196.stdout:0 2023-03-24T23:19:35.287 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0be40b23f0 msgr2=0x7f0be40af290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.287 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 --2- 172.21.15.196:0/2735631496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0be40b23f0 0x7f0be40af290 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0bd0000f60 tx=0x7f0bd000c730 comp rx=0 tx=0).stop 2023-03-24T23:19:35.288 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 shutdown_connections 2023-03-24T23:19:35.288 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 --2- 172.21.15.196:0/2735631496 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0be40ce2b0 0x7f0be40af7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.288 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 --2- 172.21.15.196:0/2735631496 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0be406acb0 0x7f0be41515a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.288 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 --2- 172.21.15.196:0/2735631496 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0be40b23f0 0x7f0be40af290 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.288 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 >> 172.21.15.196:0/2735631496 conn(0x7f0be40d0920 msgr2=0x7f0be40b2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:35.288 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 shutdown_connections 2023-03-24T23:19:35.288 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.286+0000 7f0beb182700 1 -- 172.21.15.196:0/2735631496 wait complete. 2023-03-24T23:19:35.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 55f40987-e709-41aa-84f1-7a0b426e984d 1 2023-03-24T23:19:35.371 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:35.371+0000 7f6bb5d9b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-24T23:19:35.383 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:35.382+0000 7fc54dc46200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-03-24T23:19:35.419 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.418+0000 7f3a686d3700 1 Processor -- start 2023-03-24T23:19:35.419 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.418+0000 7f3a686d3700 1 -- start start 2023-03-24T23:19:35.419 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a686d3700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a6015bc50 0x7f3a600d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.420 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a686d3700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a601588c0 con 0x7f3a600cd670 2023-03-24T23:19:35.420 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a686d3700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a600b3170 con 0x7f3a601584f0 2023-03-24T23:19:35.420 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a686d3700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a600b3420 con 0x7f3a6015bc50 2023-03-24T23:19:35.420 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a6646f700 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3a601584f0 0x7f3a600b2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:36574/0 (socket says 172.21.15.196:36574) 2023-03-24T23:19:35.420 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a65c6e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a6015bc50 0x7f3a600d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.420 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a6646f700 1 -- 172.21.15.196:0/1073029644 learned_addr learned my addr 172.21.15.196:0/1073029644 (peer_addr_for_me v1:172.21.15.196:0/0) 2023-03-24T23:19:35.422 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a6546d700 1 -- 172.21.15.196:0/1073029644 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989387084 0 0) 0x7f3a600b3170 con 0x7f3a601584f0 2023-03-24T23:19:35.422 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a6546d700 1 -- 172.21.15.196:0/1073029644 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a4c0035b0 con 0x7f3a601584f0 2023-03-24T23:19:35.422 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a65c6e700 1 -- 172.21.15.196:0/1073029644 >> v1:172.21.15.196:6789/0 conn(0x7f3a601584f0 legacy=0x7f3a600b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.423 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a65c6e700 1 -- 172.21.15.196:0/1073029644 >> v1:172.21.15.191:6789/0 conn(0x7f3a600cd670 legacy=0x7f3a600cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.423 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a65c6e700 1 -- 172.21.15.196:0/1073029644 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a600ccb90 con 0x7f3a6015bc50 2023-03-24T23:19:35.423 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a65c6e700 1 --2- 172.21.15.196:0/1073029644 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a6015bc50 0x7f3a600d00c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009870 tx=0x7f3a50009b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35bf5722f2e6d38c server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:35.423 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a6546d700 1 -- 172.21.15.196:0/1073029644 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50011830 con 0x7f3a6015bc50 2023-03-24T23:19:35.423 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a6546d700 1 -- 172.21.15.196:0/1073029644 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a50011990 con 0x7f3a6015bc50 2023-03-24T23:19:35.423 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.419+0000 7f3a6546d700 1 -- 172.21.15.196:0/1073029644 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50011d20 con 0x7f3a6015bc50 2023-03-24T23:19:35.423 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.420+0000 7f3a686d3700 1 -- 172.21.15.196:0/1073029644 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a6015bc50 msgr2=0x7f3a600d00c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.420+0000 7f3a686d3700 1 --2- 172.21.15.196:0/1073029644 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a6015bc50 0x7f3a600d00c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3a50009870 tx=0x7f3a50009b80 comp rx=0 tx=0).stop 2023-03-24T23:19:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.420+0000 7f3a686d3700 1 -- 172.21.15.196:0/1073029644 shutdown_connections 2023-03-24T23:19:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.420+0000 7f3a686d3700 1 --2- 172.21.15.196:0/1073029644 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a6015bc50 0x7f3a600d00c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.420+0000 7f3a686d3700 1 -- 172.21.15.196:0/1073029644 >> 172.21.15.196:0/1073029644 conn(0x7f3a600d0fe0 msgr2=0x7f3a600d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.420+0000 7f3a686d3700 1 -- 172.21.15.196:0/1073029644 shutdown_connections 2023-03-24T23:19:35.424 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.420+0000 7f3a686d3700 1 -- 172.21.15.196:0/1073029644 wait complete. 2023-03-24T23:19:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.421+0000 7f3a686d3700 1 Processor -- start 2023-03-24T23:19:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.421+0000 7f3a686d3700 1 -- start start 2023-03-24T23:19:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a686d3700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a600cd670 0x7f3a60151810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.426 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a686d3700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a601584f0 0x7f3a6014d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a686d3700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a601542d0 0x7f3a6006c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a686d3700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a60151d50 con 0x7f3a601542d0 2023-03-24T23:19:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a686d3700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a600b3eb0 con 0x7f3a600cd670 2023-03-24T23:19:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a686d3700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a600b4190 con 0x7f3a601584f0 2023-03-24T23:19:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a65c6e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a601584f0 0x7f3a6014d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a65c6e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a601584f0 0x7f3a6014d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:50198/0 (socket says 172.21.15.196:50198) 2023-03-24T23:19:35.427 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a65c6e700 1 -- 172.21.15.196:0/3366788912 learned_addr learned my addr 172.21.15.196:0/3366788912 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a6646f700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a600cd670 0x7f3a60151810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a66c70700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a601542d0 0x7f3a6006c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a65c6e700 1 -- 172.21.15.196:0/3366788912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a600cd670 msgr2=0x7f3a60151810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a65c6e700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a600cd670 0x7f3a60151810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a65c6e700 1 -- 172.21.15.196:0/3366788912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a601542d0 msgr2=0x7f3a6006c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a65c6e700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a601542d0 0x7f3a6006c700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.422+0000 7f3a65c6e700 1 -- 172.21.15.196:0/3366788912 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a500094a0 con 0x7f3a601584f0 2023-03-24T23:19:35.428 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.423+0000 7f3a65c6e700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a601584f0 0x7f3a6014d570 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3a500097c0 tx=0x7f3a50004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:35.429 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.423+0000 7f3a56ffd700 1 -- 172.21.15.196:0/3366788912 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50004590 con 0x7f3a601584f0 2023-03-24T23:19:35.429 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.423+0000 7f3a56ffd700 1 -- 172.21.15.196:0/3366788912 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a500036a0 con 0x7f3a601584f0 2023-03-24T23:19:35.429 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.423+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a600cd150 con 0x7f3a601584f0 2023-03-24T23:19:35.429 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.423+0000 7f3a56ffd700 1 -- 172.21.15.196:0/3366788912 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a50003a30 con 0x7f3a601584f0 2023-03-24T23:19:35.429 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.423+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a600cce70 con 0x7f3a601584f0 2023-03-24T23:19:35.429 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.423+0000 7f3a56ffd700 1 -- 172.21.15.196:0/3366788912 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3a5002e450 con 0x7f3a601584f0 2023-03-24T23:19:35.429 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.423+0000 7f3a56ffd700 1 -- 172.21.15.196:0/3366788912 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f3a5001a7d0 con 0x7f3a601584f0 2023-03-24T23:19:35.430 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.424+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a600cc7b0 con 0x7f3a601584f0 2023-03-24T23:19:35.430 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.426+0000 7f3a56ffd700 1 -- 172.21.15.196:0/3366788912 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f3a50015020 con 0x7f3a601584f0 2023-03-24T23:19:35.446 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:35.446+0000 7f6bb5d9b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-24T23:19:35.462 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:35.461+0000 7fc54dc46200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-03-24T23:19:35.511 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.510+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "osd new", "uuid": "55f40987-e709-41aa-84f1-7a0b426e984d", "id": 1} v 0) v1 -- 0x7f3a600b2e80 con 0x7f3a601584f0 2023-03-24T23:19:35.519 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.518+0000 7f3a56ffd700 1 -- 172.21.15.196:0/3366788912 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "55f40987-e709-41aa-84f1-7a0b426e984d", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3a5001aab0 con 0x7f3a601584f0 2023-03-24T23:19:35.519 INFO:teuthology.orchestra.run.smithi196.stdout:1 2023-03-24T23:19:35.520 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a601584f0 msgr2=0x7f3a6014d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.520 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a601584f0 0x7f3a6014d570 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f3a500097c0 tx=0x7f3a50004750 comp rx=0 tx=0).stop 2023-03-24T23:19:35.520 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 shutdown_connections 2023-03-24T23:19:35.521 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f3a601542d0 0x7f3a6006c700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.521 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a600cd670 0x7f3a60151810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.521 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 --2- 172.21.15.196:0/3366788912 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a601584f0 0x7f3a6014d570 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.521 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 >> 172.21.15.196:0/3366788912 conn(0x7f3a600d0fe0 msgr2=0x7f3a6006df40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:35.521 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 shutdown_connections 2023-03-24T23:19:35.521 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.519+0000 7f3a686d3700 1 -- 172.21.15.196:0/3366788912 wait complete. 2023-03-24T23:19:35.532 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new b2e378f2-4cdb-4900-8f17-83affc66d8ce 2 2023-03-24T23:19:35.569 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:35.569+0000 7f6bb5d9b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-24T23:19:35.589 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:35.588+0000 7fc54dc46200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-03-24T23:19:35.649 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:35.649+0000 7f6bb5d9b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-24T23:19:35.652 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.651+0000 7fcc1a556700 1 Processor -- start 2023-03-24T23:19:35.653 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.652+0000 7fcc1a556700 1 -- start start 2023-03-24T23:19:35.654 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc1a556700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fcc141584f0 0x7fcc140b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc1a556700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcc140b2fa0 0x7fcc140cedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc1a556700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc141588c0 con 0x7fcc1415bc50 2023-03-24T23:19:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc1a556700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc140b3420 con 0x7fcc141584f0 2023-03-24T23:19:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc1a556700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc140ccb90 con 0x7fcc140b2fa0 2023-03-24T23:19:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc18af3700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcc140b2fa0 0x7fcc140cedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc18af3700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcc140b2fa0 0x7fcc140cedc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35654/0 (socket says 172.21.15.196:35654) 2023-03-24T23:19:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc18af3700 1 -- 172.21.15.196:0/3155830295 learned_addr learned my addr 172.21.15.196:0/3155830295 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc13fff700 1 --2- 172.21.15.196:0/3155830295 >> v2:172.21.15.191:3300/0 conn(0x7fcc141584f0 0x7fcc140b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc18af3700 1 -- 172.21.15.196:0/3155830295 >> v2:172.21.15.191:3300/0 conn(0x7fcc141584f0 msgr2=0x7fcc140b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc18af3700 1 --2- 172.21.15.196:0/3155830295 >> v2:172.21.15.191:3300/0 conn(0x7fcc141584f0 0x7fcc140b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc18af3700 1 -- 172.21.15.196:0/3155830295 >> v1:172.21.15.191:6789/0 conn(0x7fcc1415bc50 legacy=0x7fcc140d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc18af3700 1 -- 172.21.15.196:0/3155830295 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc140cce70 con 0x7fcc140b2fa0 2023-03-24T23:19:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.653+0000 7fcc12ffd700 1 -- 172.21.15.196:0/3155830295 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3220147438 0 0) 0x7fcc141588c0 con 0x7fcc1415bc50 2023-03-24T23:19:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.654+0000 7fcc18af3700 1 --2- 172.21.15.196:0/3155830295 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b2fa0 0x7fcc140cedc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc140c4800 tx=0x7fcc0c00b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=408e113edbe54061 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.654+0000 7fcc12ffd700 1 -- 172.21.15.196:0/3155830295 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00e070 con 0x7fcc140b2fa0 2023-03-24T23:19:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.654+0000 7fcc12ffd700 1 -- 172.21.15.196:0/3155830295 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc0c00b910 con 0x7fcc140b2fa0 2023-03-24T23:19:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.654+0000 7fcc12ffd700 1 -- 172.21.15.196:0/3155830295 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0c00bcc0 con 0x7fcc140b2fa0 2023-03-24T23:19:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.655+0000 7fcc1a556700 1 -- 172.21.15.196:0/3155830295 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b2fa0 msgr2=0x7fcc140cedc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.655+0000 7fcc1a556700 1 --2- 172.21.15.196:0/3155830295 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b2fa0 0x7fcc140cedc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fcc140c4800 tx=0x7fcc0c00b330 comp rx=0 tx=0).stop 2023-03-24T23:19:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.655+0000 7fcc1a556700 1 -- 172.21.15.196:0/3155830295 shutdown_connections 2023-03-24T23:19:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.655+0000 7fcc1a556700 1 --2- 172.21.15.196:0/3155830295 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b2fa0 0x7fcc140cedc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.655+0000 7fcc1a556700 1 --2- 172.21.15.196:0/3155830295 >> v2:172.21.15.191:3300/0 conn(0x7fcc141584f0 0x7fcc140b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.655+0000 7fcc1a556700 1 -- 172.21.15.196:0/3155830295 >> 172.21.15.196:0/3155830295 conn(0x7fcc140d0fe0 msgr2=0x7fcc140d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.655+0000 7fcc1a556700 1 -- 172.21.15.196:0/3155830295 shutdown_connections 2023-03-24T23:19:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.655+0000 7fcc1a556700 1 -- 172.21.15.196:0/3155830295 wait complete. 2023-03-24T23:19:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.656+0000 7fcc1a556700 1 Processor -- start 2023-03-24T23:19:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.656+0000 7fcc1a556700 1 -- start start 2023-03-24T23:19:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc1a556700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcc141584f0 0x7fcc140b17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc1a556700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b3e10 0x7fcc140c3c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc1a556700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc140c41a0 0x7fcc14158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc1a556700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140c9dc0 con 0x7fcc140c41a0 2023-03-24T23:19:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc1a556700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140c4610 con 0x7fcc140b3e10 2023-03-24T23:19:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc1a556700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc141597a0 con 0x7fcc141584f0 2023-03-24T23:19:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b3e10 0x7fcc140c3c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc13fff700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcc141584f0 0x7fcc140b17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b3e10 0x7fcc140c3c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35658/0 (socket says 172.21.15.196:35658) 2023-03-24T23:19:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc13fff700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcc141584f0 0x7fcc140b17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:50208/0 (socket says 172.21.15.196:50208) 2023-03-24T23:19:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 -- 172.21.15.196:0/2770830590 learned_addr learned my addr 172.21.15.196:0/2770830590 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc18af3700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc140c41a0 0x7fcc14158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 -- 172.21.15.196:0/2770830590 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcc141584f0 msgr2=0x7fcc140b17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcc141584f0 0x7fcc140b17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 -- 172.21.15.196:0/2770830590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc140c41a0 msgr2=0x7fcc14158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc140c41a0 0x7fcc14158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 -- 172.21.15.196:0/2770830590 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c009b30 con 0x7fcc140b3e10 2023-03-24T23:19:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc13fff700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcc141584f0 0x7fcc140b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc137fe700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b3e10 0x7fcc140c3c60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800c550 tx=0x7fcc0800c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc10ff9700 1 -- 172.21.15.196:0/2770830590 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc080054e0 con 0x7fcc140b3e10 2023-03-24T23:19:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.657+0000 7fcc10ff9700 1 -- 172.21.15.196:0/2770830590 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc08003680 con 0x7fcc140b3e10 2023-03-24T23:19:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.658+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc140cc7b0 con 0x7fcc140b3e10 2023-03-24T23:19:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.658+0000 7fcc10ff9700 1 -- 172.21.15.196:0/2770830590 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc08003a30 con 0x7fcc140b3e10 2023-03-24T23:19:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.658+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc140cd150 con 0x7fcc140b3e10 2023-03-24T23:19:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.658+0000 7fcc10ff9700 1 -- 172.21.15.196:0/2770830590 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fcc0801c440 con 0x7fcc140b3e10 2023-03-24T23:19:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.658+0000 7fcc10ff9700 1 -- 172.21.15.196:0/2770830590 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7fcc08012820 con 0x7fcc140b3e10 2023-03-24T23:19:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.659+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc140cc940 con 0x7fcc140b3e10 2023-03-24T23:19:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.661+0000 7fcc10ff9700 1 -- 172.21.15.196:0/2770830590 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fcc08017070 con 0x7fcc140b3e10 2023-03-24T23:19:35.676 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:35.675+0000 7fc54dc46200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-03-24T23:19:35.746 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.744+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b2e378f2-4cdb-4900-8f17-83affc66d8ce", "id": 2} v 0) v1 -- 0x7fcc140c9f00 con 0x7fcc140b3e10 2023-03-24T23:19:35.753 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.752+0000 7fcc10ff9700 1 -- 172.21.15.196:0/2770830590 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b2e378f2-4cdb-4900-8f17-83affc66d8ce", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7fcc140c9f00 con 0x7fcc140b3e10 2023-03-24T23:19:35.753 INFO:teuthology.orchestra.run.smithi196.stdout:2 2023-03-24T23:19:35.755 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b3e10 msgr2=0x7fcc140c3c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.755 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b3e10 0x7fcc140c3c60 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcc0800c550 tx=0x7fcc0800c860 comp rx=0 tx=0).stop 2023-03-24T23:19:35.755 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 shutdown_connections 2023-03-24T23:19:35.755 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fcc140c41a0 0x7fcc14158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.755 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcc141584f0 0x7fcc140b17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.755 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 --2- 172.21.15.196:0/2770830590 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcc140b3e10 0x7fcc140c3c60 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.756 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 >> 172.21.15.196:0/2770830590 conn(0x7fcc140d0fe0 msgr2=0x7fcc140cf6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:35.756 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 shutdown_connections 2023-03-24T23:19:35.756 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.754+0000 7fcc1a556700 1 -- 172.21.15.196:0/2770830590 wait complete. 2023-03-24T23:19:35.770 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new b4483b81-cc61-477a-ada6-4620185b9f6b 3 2023-03-24T23:19:35.892 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.891+0000 7f11b378c700 1 Processor -- start 2023-03-24T23:19:35.892 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.891+0000 7f11b378c700 1 -- start start 2023-03-24T23:19:35.894 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.892+0000 7f11b378c700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f11ac159610 0x7f11ac070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.894 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.892+0000 7f11b378c700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f11ac1599e0 0x7f11ac071520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.894 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.892+0000 7f11b378c700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ac0cc530 con 0x7f11ac15e030 2023-03-24T23:19:35.894 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.892+0000 7f11b378c700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ac0cc250 con 0x7f11ac159610 2023-03-24T23:19:35.894 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.892+0000 7f11b378c700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11ac15d550 con 0x7f11ac1599e0 2023-03-24T23:19:35.895 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.892+0000 7f11b0d27700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f11ac1599e0 0x7f11ac071520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.895 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.892+0000 7f11b0d27700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f11ac1599e0 0x7f11ac071520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35664/0 (socket says 172.21.15.196:35664) 2023-03-24T23:19:35.895 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.892+0000 7f11b0d27700 1 -- 172.21.15.196:0/324168949 learned_addr learned my addr 172.21.15.196:0/324168949 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:35.895 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11b1528700 1 --2- 172.21.15.196:0/324168949 >> v2:172.21.15.191:3300/0 conn(0x7f11ac159610 0x7f11ac070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.895 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11b0d27700 1 -- 172.21.15.196:0/324168949 >> v2:172.21.15.191:3300/0 conn(0x7f11ac159610 msgr2=0x7f11ac070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:35.895 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11b0d27700 1 --2- 172.21.15.196:0/324168949 >> v2:172.21.15.191:3300/0 conn(0x7f11ac159610 0x7f11ac070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.895 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11b0d27700 1 -- 172.21.15.196:0/324168949 >> v1:172.21.15.196:6789/0 conn(0x7f11ac15e030 legacy=0x7f11ac0cb520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.896 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11b0d27700 1 -- 172.21.15.196:0/324168949 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ac15d830 con 0x7f11ac1599e0 2023-03-24T23:19:35.896 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11b0d27700 1 --2- 172.21.15.196:0/324168949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac071520 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f11ac067800 tx=0x7f119c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f92eb5d66e91253d server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:35.896 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11a3fff700 1 -- 172.21.15.196:0/324168949 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c011830 con 0x7f11ac1599e0 2023-03-24T23:19:35.896 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11a3fff700 1 -- 172.21.15.196:0/324168949 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f119c011990 con 0x7f11ac1599e0 2023-03-24T23:19:35.896 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.893+0000 7f11a3fff700 1 -- 172.21.15.196:0/324168949 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f119c011d20 con 0x7f11ac1599e0 2023-03-24T23:19:35.898 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.894+0000 7f11b378c700 1 -- 172.21.15.196:0/324168949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 msgr2=0x7f11ac071520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.894+0000 7f11b378c700 1 --2- 172.21.15.196:0/324168949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac071520 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f11ac067800 tx=0x7f119c009bd0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.894+0000 7f11b378c700 1 -- 172.21.15.196:0/324168949 shutdown_connections 2023-03-24T23:19:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.894+0000 7f11b378c700 1 --2- 172.21.15.196:0/324168949 >> v2:172.21.15.191:3300/0 conn(0x7f11ac159610 0x7f11ac070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.894+0000 7f11b378c700 1 --2- 172.21.15.196:0/324168949 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac071520 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.894+0000 7f11b378c700 1 -- 172.21.15.196:0/324168949 >> 172.21.15.196:0/324168949 conn(0x7f11ac0d0b50 msgr2=0x7f11ac0cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.894+0000 7f11b378c700 1 -- 172.21.15.196:0/324168949 shutdown_connections 2023-03-24T23:19:35.899 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.894+0000 7f11b378c700 1 -- 172.21.15.196:0/324168949 wait complete. 2023-03-24T23:19:35.900 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.895+0000 7f11b378c700 1 Processor -- start 2023-03-24T23:19:35.900 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.895+0000 7f11b378c700 1 -- start start 2023-03-24T23:19:35.900 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b378c700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac159610 0x7f11ac06ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.900 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b378c700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac151420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.900 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b378c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11ac15e030 0x7f11ac06aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:35.900 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b378c700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac06d100 con 0x7f11ac15e030 2023-03-24T23:19:35.900 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b378c700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac06d3e0 con 0x7f11ac1599e0 2023-03-24T23:19:35.900 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b378c700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11ac06d6c0 con 0x7f11ac159610 2023-03-24T23:19:35.901 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b0d27700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac151420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.901 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b1528700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac159610 0x7f11ac06ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.901 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b0d27700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac151420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35670/0 (socket says 172.21.15.196:35670) 2023-03-24T23:19:35.901 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b0d27700 1 -- 172.21.15.196:0/1609968907 learned_addr learned my addr 172.21.15.196:0/1609968907 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:35.901 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b1d29700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11ac15e030 0x7f11ac06aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:35.901 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b1528700 1 -- 172.21.15.196:0/1609968907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 msgr2=0x7f11ac151420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.901 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b1528700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac151420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.902 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b1528700 1 -- 172.21.15.196:0/1609968907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11ac15e030 msgr2=0x7f11ac06aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.902 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b1528700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11ac15e030 0x7f11ac06aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.902 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b1528700 1 -- 172.21.15.196:0/1609968907 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119c009550 con 0x7f11ac159610 2023-03-24T23:19:35.902 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.896+0000 7f11b0d27700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:35.902 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.897+0000 7f11b1528700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac159610 0x7f11ac06ca20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f11a800e900 tx=0x7f11a800ec10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:35.902 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.897+0000 7f11a1ffb700 1 -- 172.21.15.196:0/1609968907 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8013680 con 0x7f11ac159610 2023-03-24T23:19:35.903 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.897+0000 7f11a1ffb700 1 -- 172.21.15.196:0/1609968907 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a80137e0 con 0x7f11ac159610 2023-03-24T23:19:35.903 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.897+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ac0cbc30 con 0x7f11ac159610 2023-03-24T23:19:35.903 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.897+0000 7f11a1ffb700 1 -- 172.21.15.196:0/1609968907 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11a8013b50 con 0x7f11ac159610 2023-03-24T23:19:35.903 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.897+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ac15db10 con 0x7f11ac159610 2023-03-24T23:19:35.903 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.897+0000 7f11a1ffb700 1 -- 172.21.15.196:0/1609968907 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f11a8005a20 con 0x7f11ac159610 2023-03-24T23:19:35.903 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.897+0000 7f11a1ffb700 1 -- 172.21.15.196:0/1609968907 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7f11a80067a0 con 0x7f11ac159610 2023-03-24T23:19:35.904 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.898+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ac0cbdc0 con 0x7f11ac159610 2023-03-24T23:19:35.904 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.900+0000 7f11a1ffb700 1 -- 172.21.15.196:0/1609968907 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f11a8021060 con 0x7f11ac159610 2023-03-24T23:19:35.985 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.984+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b4483b81-cc61-477a-ada6-4620185b9f6b", "id": 3} v 0) v1 -- 0x7f11ac074580 con 0x7f11ac159610 2023-03-24T23:19:35.991 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.990+0000 7f11a1ffb700 1 -- 172.21.15.196:0/1609968907 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b4483b81-cc61-477a-ada6-4620185b9f6b", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7f11a8017070 con 0x7f11ac159610 2023-03-24T23:19:35.991 INFO:teuthology.orchestra.run.smithi196.stdout:3 2023-03-24T23:19:35.992 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac159610 msgr2=0x7f11ac06ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:35.992 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac159610 0x7f11ac06ca20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f11a800e900 tx=0x7f11a800ec10 comp rx=0 tx=0).stop 2023-03-24T23:19:35.992 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 shutdown_connections 2023-03-24T23:19:35.992 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f11ac15e030 0x7f11ac06aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.993 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac159610 0x7f11ac06ca20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.993 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 --2- 172.21.15.196:0/1609968907 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac1599e0 0x7f11ac151420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:35.993 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 >> 172.21.15.196:0/1609968907 conn(0x7f11ac0d0b50 msgr2=0x7f11ac0b0430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:35.993 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 shutdown_connections 2023-03-24T23:19:35.993 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:35.991+0000 7f11b378c700 1 -- 172.21.15.196:0/1609968907 wait complete. 2023-03-24T23:19:36.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new f133b7f6-ca18-4ff3-a71f-d9ae1593ce57 4 2023-03-24T23:19:36.091 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:36.090+0000 7f6bb5d9b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-24T23:19:36.124 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.123+0000 7fb92f393700 1 Processor -- start 2023-03-24T23:19:36.125 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.124+0000 7fb92f393700 1 -- start start 2023-03-24T23:19:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.124+0000 7fb92f393700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb9280cda00 0x7fb9280cde80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.126 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.124+0000 7fb92f393700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fb9280cb960 0x7fb9280cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.127 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.124+0000 7fb92f393700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9280cc2e0 con 0x7fb928159940 2023-03-24T23:19:36.127 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.124+0000 7fb92f393700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9280cb170 con 0x7fb9280cb960 2023-03-24T23:19:36.127 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.124+0000 7fb92f393700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9280cb450 con 0x7fb9280cda00 2023-03-24T23:19:36.127 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.124+0000 7fb92c92e700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb9280cda00 0x7fb9280cde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.127 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92c92e700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb9280cda00 0x7fb9280cde80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35676/0 (socket says 172.21.15.196:35676) 2023-03-24T23:19:36.127 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92c92e700 1 -- 172.21.15.196:0/17037108 learned_addr learned my addr 172.21.15.196:0/17037108 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:36.127 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92d12f700 1 --1- 172.21.15.196:0/17037108 >> v1:172.21.15.196:6789/0 conn(0x7fb928159940 0x7fb9280b4ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:36610/0 (socket says 172.21.15.196:36610) 2023-03-24T23:19:36.128 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92d930700 1 --2- 172.21.15.196:0/17037108 >> v2:172.21.15.191:3300/0 conn(0x7fb9280cb960 0x7fb9280cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.128 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92c92e700 1 -- 172.21.15.196:0/17037108 >> v2:172.21.15.191:3300/0 conn(0x7fb9280cb960 msgr2=0x7fb9280cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:36.128 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92c92e700 1 --2- 172.21.15.196:0/17037108 >> v2:172.21.15.191:3300/0 conn(0x7fb9280cb960 0x7fb9280cbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.128 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92c92e700 1 -- 172.21.15.196:0/17037108 >> v1:172.21.15.196:6789/0 conn(0x7fb928159940 legacy=0x7fb9280b4ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.128 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92c92e700 1 -- 172.21.15.196:0/17037108 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9280b42c0 con 0x7fb9280cda00 2023-03-24T23:19:36.128 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92d930700 1 --2- 172.21.15.196:0/17037108 >> v2:172.21.15.191:3300/0 conn(0x7fb9280cb960 0x7fb9280cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-24T23:19:36.128 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb92c92e700 1 --2- 172.21.15.196:0/17037108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9280cda00 0x7fb9280cde80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb928071f80 tx=0x7fb918009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=633bfa14605033d0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:36.128 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb91ffff700 1 -- 172.21.15.196:0/17037108 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800b870 con 0x7fb9280cda00 2023-03-24T23:19:36.129 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb91ffff700 1 -- 172.21.15.196:0/17037108 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb91800b9d0 con 0x7fb9280cda00 2023-03-24T23:19:36.129 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.125+0000 7fb91ffff700 1 -- 172.21.15.196:0/17037108 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb91800bd60 con 0x7fb9280cda00 2023-03-24T23:19:36.131 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.126+0000 7fb92f393700 1 -- 172.21.15.196:0/17037108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9280cda00 msgr2=0x7fb9280cde80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.131 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.126+0000 7fb92f393700 1 --2- 172.21.15.196:0/17037108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9280cda00 0x7fb9280cde80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb928071f80 tx=0x7fb918009bd0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.131 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.126+0000 7fb92f393700 1 -- 172.21.15.196:0/17037108 shutdown_connections 2023-03-24T23:19:36.131 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.126+0000 7fb92f393700 1 --2- 172.21.15.196:0/17037108 >> v2:172.21.15.191:3300/0 conn(0x7fb9280cb960 0x7fb9280cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.132 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.126+0000 7fb92f393700 1 --2- 172.21.15.196:0/17037108 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9280cda00 0x7fb9280cde80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.132 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.126+0000 7fb92f393700 1 -- 172.21.15.196:0/17037108 >> 172.21.15.196:0/17037108 conn(0x7fb9280d0b50 msgr2=0x7fb9280cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:36.132 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.126+0000 7fb92f393700 1 -- 172.21.15.196:0/17037108 shutdown_connections 2023-03-24T23:19:36.132 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.126+0000 7fb92f393700 1 -- 172.21.15.196:0/17037108 wait complete. 2023-03-24T23:19:36.132 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.128+0000 7fb92f393700 1 Processor -- start 2023-03-24T23:19:36.132 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.128+0000 7fb92f393700 1 -- start start 2023-03-24T23:19:36.132 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.128+0000 7fb92f393700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9280cb960 0x7fb9281ef600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.133 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.128+0000 7fb92f393700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9280cda00 0x7fb9280b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.133 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.128+0000 7fb92f393700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb928159940 0x7fb92815bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.133 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.128+0000 7fb92f393700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280b2940 con 0x7fb9280cb960 2023-03-24T23:19:36.133 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92f393700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281efb40 con 0x7fb928159940 2023-03-24T23:19:36.133 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92f393700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9280b20a0 con 0x7fb9280cda00 2023-03-24T23:19:36.133 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb928159940 0x7fb92815bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.133 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb928159940 0x7fb92815bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35684/0 (socket says 172.21.15.196:35684) 2023-03-24T23:19:36.134 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 -- 172.21.15.196:0/1506642247 learned_addr learned my addr 172.21.15.196:0/1506642247 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:36.134 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d12f700 1 --2- 172.21.15.196:0/1506642247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9280cb960 0x7fb9281ef600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.134 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 -- 172.21.15.196:0/1506642247 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9280cda00 msgr2=0x7fb9280b1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.134 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 --2- 172.21.15.196:0/1506642247 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9280cda00 0x7fb9280b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.134 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 -- 172.21.15.196:0/1506642247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9280cb960 msgr2=0x7fb9281ef600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.134 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 --2- 172.21.15.196:0/1506642247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9280cb960 0x7fb9281ef600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.134 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 -- 172.21.15.196:0/1506642247 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb918009550 con 0x7fb928159940 2023-03-24T23:19:36.134 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92d930700 1 --2- 172.21.15.196:0/1506642247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb928159940 0x7fb92815bf70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb920007df0 tx=0x7fb92000d3b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:36.135 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb91dffb700 1 -- 172.21.15.196:0/1506642247 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb92000ddd0 con 0x7fb928159940 2023-03-24T23:19:36.135 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb91dffb700 1 -- 172.21.15.196:0/1506642247 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb92000f040 con 0x7fb928159940 2023-03-24T23:19:36.135 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9280b51d0 con 0x7fb928159940 2023-03-24T23:19:36.135 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb91dffb700 1 -- 172.21.15.196:0/1506642247 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb920003c60 con 0x7fb928159940 2023-03-24T23:19:36.135 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.129+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9280b45a0 con 0x7fb928159940 2023-03-24T23:19:36.135 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.130+0000 7fb91dffb700 1 -- 172.21.15.196:0/1506642247 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb920004330 con 0x7fb928159940 2023-03-24T23:19:36.135 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.130+0000 7fb91dffb700 1 -- 172.21.15.196:0/1506642247 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7fb920015950 con 0x7fb928159940 2023-03-24T23:19:36.135 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.130+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9280b3f00 con 0x7fb928159940 2023-03-24T23:19:36.136 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.132+0000 7fb91dffb700 1 -- 172.21.15.196:0/1506642247 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fb92001a070 con 0x7fb928159940 2023-03-24T23:19:36.149 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:36.148+0000 7fc54dc46200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-03-24T23:19:36.217 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.216+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f133b7f6-ca18-4ff3-a71f-d9ae1593ce57", "id": 4} v 0) v1 -- 0x7fb9280b2fc0 con 0x7fb928159940 2023-03-24T23:19:36.223 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.222+0000 7fb91dffb700 1 -- 172.21.15.196:0/1506642247 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f133b7f6-ca18-4ff3-a71f-d9ae1593ce57", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb920003dc0 con 0x7fb928159940 2023-03-24T23:19:36.223 INFO:teuthology.orchestra.run.smithi196.stdout:4 2023-03-24T23:19:36.224 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb928159940 msgr2=0x7fb92815bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.224 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 --2- 172.21.15.196:0/1506642247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb928159940 0x7fb92815bf70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb920007df0 tx=0x7fb92000d3b0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.224 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 shutdown_connections 2023-03-24T23:19:36.224 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 --2- 172.21.15.196:0/1506642247 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fb9280cb960 0x7fb9281ef600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.225 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 --2- 172.21.15.196:0/1506642247 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb928159940 0x7fb92815bf70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.225 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 --2- 172.21.15.196:0/1506642247 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9280cda00 0x7fb9280b1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.225 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 >> 172.21.15.196:0/1506642247 conn(0x7fb9280d0b50 msgr2=0x7fb9280c3960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:36.225 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 shutdown_connections 2023-03-24T23:19:36.225 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.223+0000 7fb92f393700 1 -- 172.21.15.196:0/1506642247 wait complete. 2023-03-24T23:19:36.236 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new b284d515-2796-4d33-81a8-0e42195d49bb 5 2023-03-24T23:19:36.356 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.355+0000 7fc4e1b97700 1 Processor -- start 2023-03-24T23:19:36.356 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.355+0000 7fc4e1b97700 1 -- start start 2023-03-24T23:19:36.358 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4e1b97700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc4dc1584f0 0x7fc4dc0b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.359 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4e1b97700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc4dc15bc50 0x7fc4dc0d00c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.359 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4e1b97700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc0b2fa0 0x7fc4dc0d0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.359 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4e1b97700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4dc1588c0 con 0x7fc4dc1584f0 2023-03-24T23:19:36.359 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4e1b97700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4dc0b3420 con 0x7fc4dc15bc50 2023-03-24T23:19:36.359 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4e1b97700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0ccb90 con 0x7fc4dc0b2fa0 2023-03-24T23:19:36.359 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4daffd700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc4dc15bc50 0x7fc4dc0d00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4dbfff700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc0b2fa0 0x7fc4dc0d0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4daffd700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc4dc15bc50 0x7fc4dc0d00c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35688/0 (socket says 172.21.15.196:35688) 2023-03-24T23:19:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4daffd700 1 -- 172.21.15.196:0/3101993735 learned_addr learned my addr 172.21.15.196:0/3101993735 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4db7fe700 1 --2- 172.21.15.196:0/3101993735 >> v2:172.21.15.191:3300/0 conn(0x7fc4dc1584f0 0x7fc4dc0b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4dbfff700 1 -- 172.21.15.196:0/3101993735 >> v2:172.21.15.196:3300/0 conn(0x7fc4dc15bc50 msgr2=0x7fc4dc0d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4dbfff700 1 --2- 172.21.15.196:0/3101993735 >> v2:172.21.15.196:3300/0 conn(0x7fc4dc15bc50 0x7fc4dc0d00c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4dbfff700 1 -- 172.21.15.196:0/3101993735 >> v2:172.21.15.191:3300/0 conn(0x7fc4dc1584f0 msgr2=0x7fc4dc0b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:36.360 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4dbfff700 1 --2- 172.21.15.196:0/3101993735 >> v2:172.21.15.191:3300/0 conn(0x7fc4dc1584f0 0x7fc4dc0b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.361 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4dbfff700 1 -- 172.21.15.196:0/3101993735 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4dc0cce70 con 0x7fc4dc0b2fa0 2023-03-24T23:19:36.361 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.356+0000 7fc4daffd700 1 --2- 172.21.15.196:0/3101993735 >> v2:172.21.15.196:3300/0 conn(0x7fc4dc15bc50 0x7fc4dc0d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:36.361 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4dbfff700 1 --2- 172.21.15.196:0/3101993735 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc0b2fa0 0x7fc4dc0d0600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0b09f0 tx=0x7fc4d400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be5ff4c8a2026d92 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:36.361 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4da7fc700 1 -- 172.21.15.196:0/3101993735 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d400e070 con 0x7fc4dc0b2fa0 2023-03-24T23:19:36.361 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4da7fc700 1 -- 172.21.15.196:0/3101993735 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4d400bb20 con 0x7fc4dc0b2fa0 2023-03-24T23:19:36.361 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4da7fc700 1 -- 172.21.15.196:0/3101993735 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4012430 con 0x7fc4dc0b2fa0 2023-03-24T23:19:36.361 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 -- 172.21.15.196:0/3101993735 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc0b2fa0 msgr2=0x7fc4dc0d0600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.362 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 --2- 172.21.15.196:0/3101993735 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc0b2fa0 0x7fc4dc0d0600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0b09f0 tx=0x7fc4d400b330 comp rx=0 tx=0).stop 2023-03-24T23:19:36.362 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 -- 172.21.15.196:0/3101993735 shutdown_connections 2023-03-24T23:19:36.362 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 --2- 172.21.15.196:0/3101993735 >> v2:172.21.15.191:3300/0 conn(0x7fc4dc1584f0 0x7fc4dc0b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.362 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 --2- 172.21.15.196:0/3101993735 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc0b2fa0 0x7fc4dc0d0600 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.362 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 --2- 172.21.15.196:0/3101993735 >> v2:172.21.15.196:3300/0 conn(0x7fc4dc15bc50 0x7fc4dc0d00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.362 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 -- 172.21.15.196:0/3101993735 >> 172.21.15.196:0/3101993735 conn(0x7fc4dc0d0fe0 msgr2=0x7fc4dc0d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:36.362 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 -- 172.21.15.196:0/3101993735 shutdown_connections 2023-03-24T23:19:36.362 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.357+0000 7fc4e1b97700 1 -- 172.21.15.196:0/3101993735 wait complete. 2023-03-24T23:19:36.364 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.359+0000 7fc4e1b97700 1 Processor -- start 2023-03-24T23:19:36.364 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.359+0000 7fc4e1b97700 1 -- start start 2023-03-24T23:19:36.365 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.359+0000 7fc4e1b97700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4dc1584f0 0x7fc4dc0c5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.365 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.359+0000 7fc4e1b97700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc1ef950 0x7fc4dc1e4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.365 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.359+0000 7fc4e1b97700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4dc1e4ac0 0x7fc4dc1f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.365 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.359+0000 7fc4e1b97700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0b2b40 con 0x7fc4dc1e4ac0 2023-03-24T23:19:36.365 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.359+0000 7fc4e1b97700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc1e4f60 con 0x7fc4dc1584f0 2023-03-24T23:19:36.365 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.359+0000 7fc4e1b97700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4dc0c5d00 con 0x7fc4dc1ef950 2023-03-24T23:19:36.365 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4dc1584f0 0x7fc4dc0c5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.366 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4dc1584f0 0x7fc4dc0c5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35692/0 (socket says 172.21.15.196:35692) 2023-03-24T23:19:36.366 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 -- 172.21.15.196:0/579801204 learned_addr learned my addr 172.21.15.196:0/579801204 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:36.366 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4daffd700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc1ef950 0x7fc4dc1e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.366 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4dbfff700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4dc1e4ac0 0x7fc4dc1f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.366 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 -- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc1ef950 msgr2=0x7fc4dc1e4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.366 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc1ef950 0x7fc4dc1e4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.366 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 -- 172.21.15.196:0/579801204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4dc1e4ac0 msgr2=0x7fc4dc1f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.367 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4dc1e4ac0 0x7fc4dc1f5bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.367 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 -- 172.21.15.196:0/579801204 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d0009550 con 0x7fc4dc1584f0 2023-03-24T23:19:36.367 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4daffd700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc1ef950 0x7fc4dc1e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:36.367 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4dbfff700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4dc1e4ac0 0x7fc4dc1f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:36.367 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4db7fe700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4dc1584f0 0x7fc4dc0c5230 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00e960 tx=0x7fc4cc00ec70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:36.367 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4c3fff700 1 -- 172.21.15.196:0/579801204 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc013680 con 0x7fc4dc1584f0 2023-03-24T23:19:36.367 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4c3fff700 1 -- 172.21.15.196:0/579801204 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4cc0137e0 con 0x7fc4dc1584f0 2023-03-24T23:19:36.367 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d4009a70 con 0x7fc4dc1584f0 2023-03-24T23:19:36.368 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4c3fff700 1 -- 172.21.15.196:0/579801204 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc013b70 con 0x7fc4dc1584f0 2023-03-24T23:19:36.368 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4dc0cc7b0 con 0x7fc4dc1584f0 2023-03-24T23:19:36.368 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4c3fff700 1 -- 172.21.15.196:0/579801204 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc4cc005a20 con 0x7fc4dc1584f0 2023-03-24T23:19:36.368 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.360+0000 7fc4c3fff700 1 -- 172.21.15.196:0/579801204 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7fc4cc006700 con 0x7fc4dc1584f0 2023-03-24T23:19:36.368 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.362+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4dc0cd150 con 0x7fc4dc1584f0 2023-03-24T23:19:36.368 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.364+0000 7fc4c3fff700 1 -- 172.21.15.196:0/579801204 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fc4cc021060 con 0x7fc4dc1584f0 2023-03-24T23:19:36.448 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.447+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b284d515-2796-4d33-81a8-0e42195d49bb", "id": 5} v 0) v1 -- 0x7fc4dc0cc940 con 0x7fc4dc1584f0 2023-03-24T23:19:36.456 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.455+0000 7fc4c3fff700 1 -- 172.21.15.196:0/579801204 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b284d515-2796-4d33-81a8-0e42195d49bb", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc4cc017070 con 0x7fc4dc1584f0 2023-03-24T23:19:36.456 INFO:teuthology.orchestra.run.smithi196.stdout:5 2023-03-24T23:19:36.457 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4dc1584f0 msgr2=0x7fc4dc0c5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.457 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4dc1584f0 0x7fc4dc0c5230 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00e960 tx=0x7fc4cc00ec70 comp rx=0 tx=0).stop 2023-03-24T23:19:36.457 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 shutdown_connections 2023-03-24T23:19:36.457 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4dc1e4ac0 0x7fc4dc1f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.457 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4dc1584f0 0x7fc4dc0c5230 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.458 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 --2- 172.21.15.196:0/579801204 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4dc1ef950 0x7fc4dc1e4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.458 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 >> 172.21.15.196:0/579801204 conn(0x7fc4dc0d0fe0 msgr2=0x7fc4dc0cd670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:36.458 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 shutdown_connections 2023-03-24T23:19:36.458 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.456+0000 7fc4e1b97700 1 -- 172.21.15.196:0/579801204 wait complete. 2023-03-24T23:19:36.468 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 37ee78d6-7798-4d17-ad11-4fffd8355d73 6 2023-03-24T23:19:36.616 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.615+0000 7f65f6554700 1 Processor -- start 2023-03-24T23:19:36.616 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.615+0000 7f65f6554700 1 -- start start 2023-03-24T23:19:36.618 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65f6554700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f65f015a830 0x7f65f00cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.618 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65f6554700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f00b3170 0x7f65f00cf320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.618 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65f6554700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65f00cebb0 con 0x7f65f015e310 2023-03-24T23:19:36.618 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65f6554700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65f015ac00 con 0x7f65f015a830 2023-03-24T23:19:36.619 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65f6554700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00cd150 con 0x7f65f00b3170 2023-03-24T23:19:36.619 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65effff700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f65f015a830 0x7f65f00cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.619 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65f4af1700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f00b3170 0x7f65f00cf320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.619 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65effff700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f65f015a830 0x7f65f00cc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35698/0 (socket says 172.21.15.196:35698) 2023-03-24T23:19:36.619 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.616+0000 7f65effff700 1 -- 172.21.15.196:0/3975897255 learned_addr learned my addr 172.21.15.196:0/3975897255 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:36.619 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65effff700 1 -- 172.21.15.196:0/3975897255 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f00b3170 msgr2=0x7f65f00cf320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:36.619 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65effff700 1 --2- 172.21.15.196:0/3975897255 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f00b3170 0x7f65f00cf320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.620 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65effff700 1 -- 172.21.15.196:0/3975897255 >> v1:172.21.15.191:6789/0 conn(0x7f65f015e310 legacy=0x7f65f00b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.620 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65effff700 1 -- 172.21.15.196:0/3975897255 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65f00cd430 con 0x7f65f015a830 2023-03-24T23:19:36.620 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65f4af1700 1 --2- 172.21.15.196:0/3975897255 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f00b3170 0x7f65f00cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:36.620 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65effff700 1 --2- 172.21.15.196:0/3975897255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015a830 0x7f65f00cc820 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f65e00098f0 tx=0x7f65e0009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4981dcdab3c200c server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:36.620 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65eeffd700 1 -- 172.21.15.196:0/3975897255 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0011830 con 0x7f65f015a830 2023-03-24T23:19:36.620 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65eeffd700 1 -- 172.21.15.196:0/3975897255 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e0011990 con 0x7f65f015a830 2023-03-24T23:19:36.620 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.617+0000 7f65eeffd700 1 -- 172.21.15.196:0/3975897255 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e0011d20 con 0x7f65f015a830 2023-03-24T23:19:36.623 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.618+0000 7f65f6554700 1 -- 172.21.15.196:0/3975897255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015a830 msgr2=0x7f65f00cc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.623 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.618+0000 7f65f6554700 1 --2- 172.21.15.196:0/3975897255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015a830 0x7f65f00cc820 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f65e00098f0 tx=0x7f65e0009c00 comp rx=0 tx=0).stop 2023-03-24T23:19:36.623 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.618+0000 7f65f6554700 1 -- 172.21.15.196:0/3975897255 shutdown_connections 2023-03-24T23:19:36.623 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.618+0000 7f65f6554700 1 --2- 172.21.15.196:0/3975897255 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f00b3170 0x7f65f00cf320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.623 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.618+0000 7f65f6554700 1 --2- 172.21.15.196:0/3975897255 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015a830 0x7f65f00cc820 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.623 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.618+0000 7f65f6554700 1 -- 172.21.15.196:0/3975897255 >> 172.21.15.196:0/3975897255 conn(0x7f65f00d0fe0 msgr2=0x7f65f00d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:36.624 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.618+0000 7f65f6554700 1 -- 172.21.15.196:0/3975897255 shutdown_connections 2023-03-24T23:19:36.624 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.618+0000 7f65f6554700 1 -- 172.21.15.196:0/3975897255 wait complete. 2023-03-24T23:19:36.624 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.619+0000 7f65f6554700 1 Processor -- start 2023-03-24T23:19:36.624 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.619+0000 7f65f6554700 1 -- start start 2023-03-24T23:19:36.624 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f6554700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f00b3170 0x7f65f0155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.624 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f6554700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f015a830 0x7f65f014b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.624 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f6554700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015e310 0x7f65f00c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f6554700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00c9a90 con 0x7f65f00b3170 2023-03-24T23:19:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f6554700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65f00b6e10 con 0x7f65f015e310 2023-03-24T23:19:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f6554700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65f01f0520 con 0x7f65f015a830 2023-03-24T23:19:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f4af1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015e310 0x7f65f00c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f015a830 0x7f65f014b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f015a830 0x7f65f014b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:50258/0 (socket says 172.21.15.196:50258) 2023-03-24T23:19:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f4af1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015e310 0x7f65f00c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:35708/0 (socket says 172.21.15.196:35708) 2023-03-24T23:19:36.625 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 -- 172.21.15.196:0/2258307226 learned_addr learned my addr 172.21.15.196:0/2258307226 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:36.626 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65effff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f00b3170 0x7f65f0155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.626 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 -- 172.21.15.196:0/2258307226 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015e310 msgr2=0x7f65f00c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.626 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015e310 0x7f65f00c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.626 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 -- 172.21.15.196:0/2258307226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f00b3170 msgr2=0x7f65f0155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.626 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f00b3170 0x7f65f0155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.626 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 -- 172.21.15.196:0/2258307226 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e8009a70 con 0x7f65f015a830 2023-03-24T23:19:36.626 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65f4af1700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015e310 0x7f65f00c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65effff700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f00b3170 0x7f65f0155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ef7fe700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f015a830 0x7f65f014b420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f65e400c450 tx=0x7f65e400c760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.620+0000 7f65ecff9700 1 -- 172.21.15.196:0/2258307226 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e40052d0 con 0x7f65f015a830 2023-03-24T23:19:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.621+0000 7f65ecff9700 1 -- 172.21.15.196:0/2258307226 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e4005430 con 0x7f65f015a830 2023-03-24T23:19:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.621+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e0009550 con 0x7f65f015a830 2023-03-24T23:19:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.621+0000 7f65ecff9700 1 -- 172.21.15.196:0/2258307226 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65e40038c0 con 0x7f65f015a830 2023-03-24T23:19:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.621+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65f00cee00 con 0x7f65f015a830 2023-03-24T23:19:36.627 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.621+0000 7f65ecff9700 1 -- 172.21.15.196:0/2258307226 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f65e4003d20 con 0x7f65f015a830 2023-03-24T23:19:36.628 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.621+0000 7f65ecff9700 1 -- 172.21.15.196:0/2258307226 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7f65e4012a10 con 0x7f65f015a830 2023-03-24T23:19:36.628 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.622+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65f00cd710 con 0x7f65f015a830 2023-03-24T23:19:36.628 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.624+0000 7f65ecff9700 1 -- 172.21.15.196:0/2258307226 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f65e4017070 con 0x7f65f015a830 2023-03-24T23:19:36.650 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:36.650+0000 7f6bb5d9b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-24T23:19:36.708 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.707+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "osd new", "uuid": "37ee78d6-7798-4d17-ad11-4fffd8355d73", "id": 6} v 0) v1 -- 0x7f65f015f200 con 0x7f65f015a830 2023-03-24T23:19:36.715 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.715+0000 7f65ecff9700 1 -- 172.21.15.196:0/2258307226 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "37ee78d6-7798-4d17-ad11-4fffd8355d73", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f65e4012420 con 0x7f65f015a830 2023-03-24T23:19:36.716 INFO:teuthology.orchestra.run.smithi196.stdout:6 2023-03-24T23:19:36.717 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f015a830 msgr2=0x7f65f014b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.717 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f015a830 0x7f65f014b420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f65e400c450 tx=0x7f65e400c760 comp rx=0 tx=0).stop 2023-03-24T23:19:36.717 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 shutdown_connections 2023-03-24T23:19:36.717 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f65f00b3170 0x7f65f0155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.717 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65f015e310 0x7f65f00c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.718 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 --2- 172.21.15.196:0/2258307226 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65f015a830 0x7f65f014b420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.718 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 >> 172.21.15.196:0/2258307226 conn(0x7f65f00d0fe0 msgr2=0x7f65f00662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:36.718 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 shutdown_connections 2023-03-24T23:19:36.718 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.716+0000 7f65f6554700 1 -- 172.21.15.196:0/2258307226 wait complete. 2023-03-24T23:19:36.729 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:36.727+0000 7fc54dc46200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-03-24T23:19:36.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 0c1c2708-0d59-4f5b-89aa-75ea67df828e 7 2023-03-24T23:19:36.807 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:36.807+0000 7f6bb5d9b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-24T23:19:36.853 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.852+0000 7fc4a9aff700 1 Processor -- start 2023-03-24T23:19:36.854 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.852+0000 7fc4a9aff700 1 -- start start 2023-03-24T23:19:36.855 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a9aff700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415b740 0x7fc4a415e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.855 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a9aff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fc4a40cb900 0x7fc4a4070860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.855 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a9aff700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4a40b26b0 con 0x7fc4a415ac60 2023-03-24T23:19:36.855 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a9aff700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4a415f6b0 con 0x7fc4a40cb900 2023-03-24T23:19:36.856 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a9aff700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40cafa0 con 0x7fc4a415b740 2023-03-24T23:19:36.856 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a37fe700 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc4a415ac60 0x7fc4a415b030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:36646/0 (socket says 172.21.15.196:36646) 2023-03-24T23:19:36.856 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a2ffd700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415b740 0x7fc4a415e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.856 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a37fe700 1 -- 172.21.15.196:0/1828716348 learned_addr learned my addr 172.21.15.196:0/1828716348 (peer_addr_for_me v1:172.21.15.196:0/0) 2023-03-24T23:19:36.856 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.853+0000 7fc4a3fff700 1 --2- 172.21.15.196:0/1828716348 >> v2:172.21.15.191:3300/0 conn(0x7fc4a40cb900 0x7fc4a4070860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.856 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a27fc700 1 -- 172.21.15.196:0/1828716348 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 709535924 0 0) 0x7fc4a40b26b0 con 0x7fc4a415ac60 2023-03-24T23:19:36.856 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a2ffd700 1 -- 172.21.15.196:0/1828716348 >> v2:172.21.15.191:3300/0 conn(0x7fc4a40cb900 msgr2=0x7fc4a4070860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:36.857 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a2ffd700 1 --2- 172.21.15.196:0/1828716348 >> v2:172.21.15.191:3300/0 conn(0x7fc4a40cb900 0x7fc4a4070860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.857 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a2ffd700 1 -- 172.21.15.196:0/1828716348 >> v1:172.21.15.196:6789/0 conn(0x7fc4a415ac60 legacy=0x7fc4a415b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.857 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a2ffd700 1 -- 172.21.15.196:0/1828716348 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a415ee10 con 0x7fc4a415b740 2023-03-24T23:19:36.857 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a2ffd700 1 --2- 172.21.15.196:0/1828716348 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415b740 0x7fc4a415e730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009840 tx=0x7fc48c009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c6a0df22bcb283b4 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:36.857 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a27fc700 1 -- 172.21.15.196:0/1828716348 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c011830 con 0x7fc4a415b740 2023-03-24T23:19:36.858 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a27fc700 1 -- 172.21.15.196:0/1828716348 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c011990 con 0x7fc4a415b740 2023-03-24T23:19:36.858 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.854+0000 7fc4a27fc700 1 -- 172.21.15.196:0/1828716348 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c011d20 con 0x7fc4a415b740 2023-03-24T23:19:36.860 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.855+0000 7fc4a9aff700 1 -- 172.21.15.196:0/1828716348 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415b740 msgr2=0x7fc4a415e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.860 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.855+0000 7fc4a9aff700 1 --2- 172.21.15.196:0/1828716348 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415b740 0x7fc4a415e730 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009840 tx=0x7fc48c009b50 comp rx=0 tx=0).stop 2023-03-24T23:19:36.860 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.855+0000 7fc4a9aff700 1 -- 172.21.15.196:0/1828716348 shutdown_connections 2023-03-24T23:19:36.860 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.855+0000 7fc4a9aff700 1 --2- 172.21.15.196:0/1828716348 >> v2:172.21.15.191:3300/0 conn(0x7fc4a40cb900 0x7fc4a4070860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.855+0000 7fc4a9aff700 1 --2- 172.21.15.196:0/1828716348 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415b740 0x7fc4a415e730 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.855+0000 7fc4a9aff700 1 -- 172.21.15.196:0/1828716348 >> 172.21.15.196:0/1828716348 conn(0x7fc4a40ccd40 msgr2=0x7fc4a40cf630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.855+0000 7fc4a9aff700 1 -- 172.21.15.196:0/1828716348 shutdown_connections 2023-03-24T23:19:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.855+0000 7fc4a9aff700 1 -- 172.21.15.196:0/1828716348 wait complete. 2023-03-24T23:19:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.856+0000 7fc4a9aff700 1 Processor -- start 2023-03-24T23:19:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.856+0000 7fc4a9aff700 1 -- start start 2023-03-24T23:19:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a9aff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4a40cb900 0x7fc4a40afae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.861 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a9aff700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415ac60 0x7fc4a4148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.862 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a9aff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4a415b740 0x7fc4a40c5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:36.862 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a9aff700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40af650 con 0x7fc4a40cb900 2023-03-24T23:19:36.862 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a9aff700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40b0020 con 0x7fc4a415b740 2023-03-24T23:19:36.862 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a9aff700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a40c5b80 con 0x7fc4a415ac60 2023-03-24T23:19:36.862 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415ac60 0x7fc4a4148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.862 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a3fff700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4a415b740 0x7fc4a40c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.862 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415ac60 0x7fc4a4148a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:50270/0 (socket says 172.21.15.196:50270) 2023-03-24T23:19:36.863 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 -- 172.21.15.196:0/143985650 learned_addr learned my addr 172.21.15.196:0/143985650 (peer_addr_for_me v2:172.21.15.196:0/0) 2023-03-24T23:19:36.863 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a37fe700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4a40cb900 0x7fc4a40afae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:36.863 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 -- 172.21.15.196:0/143985650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4a415b740 msgr2=0x7fc4a40c5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.863 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4a415b740 0x7fc4a40c5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.863 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 -- 172.21.15.196:0/143985650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4a40cb900 msgr2=0x7fc4a40afae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.863 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4a40cb900 0x7fc4a40afae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.863 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 -- 172.21.15.196:0/143985650 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c0094a0 con 0x7fc4a415ac60 2023-03-24T23:19:36.864 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a3fff700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4a415b740 0x7fc4a40c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:36.864 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc4a2ffd700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415ac60 0x7fc4a4148a40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009fd0 tx=0x7fc48c004750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:36.864 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.857+0000 7fc49bfff700 1 -- 172.21.15.196:0/143985650 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c0036a0 con 0x7fc4a415ac60 2023-03-24T23:19:36.864 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.858+0000 7fc49bfff700 1 -- 172.21.15.196:0/143985650 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c01e070 con 0x7fc4a415ac60 2023-03-24T23:19:36.864 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.858+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a415f3d0 con 0x7fc4a415ac60 2023-03-24T23:19:36.864 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.858+0000 7fc49bfff700 1 -- 172.21.15.196:0/143985650 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c003a60 con 0x7fc4a415ac60 2023-03-24T23:19:36.864 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.858+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a415f0f0 con 0x7fc4a415ac60 2023-03-24T23:19:36.865 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.858+0000 7fc49bfff700 1 -- 172.21.15.196:0/143985650 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fc48c02d450 con 0x7fc4a415ac60 2023-03-24T23:19:36.865 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.858+0000 7fc49bfff700 1 -- 172.21.15.196:0/143985650 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7fc48c01a420 con 0x7fc4a415ac60 2023-03-24T23:19:36.865 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.858+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a4070da0 con 0x7fc4a415ac60 2023-03-24T23:19:36.865 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.861+0000 7fc49bfff700 1 -- 172.21.15.196:0/143985650 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7fc48c020070 con 0x7fc4a415ac60 2023-03-24T23:19:36.893 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:36.892+0000 7fc54dc46200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-03-24T23:19:36.945 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.944+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0c1c2708-0d59-4f5b-89aa-75ea67df828e", "id": 7} v 0) v1 -- 0x7fc4a40d0780 con 0x7fc4a415ac60 2023-03-24T23:19:36.953 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.952+0000 7fc49bfff700 1 -- 172.21.15.196:0/143985650 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0c1c2708-0d59-4f5b-89aa-75ea67df828e", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc48c015020 con 0x7fc4a415ac60 2023-03-24T23:19:36.953 INFO:teuthology.orchestra.run.smithi196.stdout:7 2023-03-24T23:19:36.954 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415ac60 msgr2=0x7fc4a4148a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:36.955 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415ac60 0x7fc4a4148a40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc48c009fd0 tx=0x7fc48c004750 comp rx=0 tx=0).stop 2023-03-24T23:19:36.955 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 shutdown_connections 2023-03-24T23:19:36.955 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fc4a40cb900 0x7fc4a40afae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.955 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4a415b740 0x7fc4a40c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.955 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 --2- 172.21.15.196:0/143985650 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4a415ac60 0x7fc4a4148a40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:36.955 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 >> 172.21.15.196:0/143985650 conn(0x7fc4a40ccd40 msgr2=0x7fc4a40b2b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:36.955 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 shutdown_connections 2023-03-24T23:19:36.956 INFO:teuthology.orchestra.run.smithi196.stderr:2023-03-24T23:19:36.953+0000 7fc4a9aff700 1 -- 172.21.15.196:0/143985650 wait complete. 2023-03-24T23:19:36.966 INFO:tasks.ceph.osd.0:Restarting daemon 2023-03-24T23:19:36.967 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-03-24T23:19:36.969 INFO:tasks.ceph.osd.0:Started 2023-03-24T23:19:36.970 INFO:tasks.ceph.osd.1:Restarting daemon 2023-03-24T23:19:36.970 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-03-24T23:19:36.972 INFO:tasks.ceph.osd.1:Started 2023-03-24T23:19:36.972 INFO:tasks.ceph.osd.2:Restarting daemon 2023-03-24T23:19:36.972 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-03-24T23:19:36.974 INFO:tasks.ceph.osd.2:Started 2023-03-24T23:19:36.974 INFO:tasks.ceph.osd.3:Restarting daemon 2023-03-24T23:19:36.975 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-03-24T23:19:36.977 INFO:tasks.ceph.osd.3:Started 2023-03-24T23:19:36.977 INFO:tasks.ceph.osd.4:Restarting daemon 2023-03-24T23:19:36.977 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-03-24T23:19:36.979 INFO:tasks.ceph.osd.4:Started 2023-03-24T23:19:36.979 INFO:tasks.ceph.osd.5:Restarting daemon 2023-03-24T23:19:36.980 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-03-24T23:19:36.981 INFO:tasks.ceph.osd.5:Started 2023-03-24T23:19:36.982 INFO:tasks.ceph.osd.6:Restarting daemon 2023-03-24T23:19:36.982 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-03-24T23:19:36.984 INFO:tasks.ceph.osd.6:Started 2023-03-24T23:19:36.984 INFO:tasks.ceph.osd.7:Restarting daemon 2023-03-24T23:19:36.984 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-03-24T23:19:36.986 INFO:tasks.ceph.osd.7:Started 2023-03-24T23:19:36.987 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-24T23:19:37.159 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.158+0000 7f75043ab700 1 Processor -- start 2023-03-24T23:19:37.159 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.158+0000 7f75043ab700 1 -- start start 2023-03-24T23:19:37.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.159+0000 7f75043ab700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f74fc0ceff0 0x7f74fc06b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:37.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.159+0000 7f75043ab700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc0b1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:37.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.159+0000 7f75043ab700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74fc06c580 con 0x7f74fc0cb8d0 2023-03-24T23:19:37.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.159+0000 7f75043ab700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74fc0b1ab0 con 0x7f74fc0ceff0 2023-03-24T23:19:37.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.159+0000 7f75043ab700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc160a20 con 0x7f74fc0b10f0 2023-03-24T23:19:37.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.159+0000 7f7502147700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f74fc0cb8d0 0x7f74fc160eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57410/0 (socket says 172.21.15.191:57410) 2023-03-24T23:19:37.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.159+0000 7f7502147700 1 -- 172.21.15.191:0/787853112 learned_addr learned my addr 172.21.15.191:0/787853112 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:37.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501946700 1 --2- 172.21.15.191:0/787853112 >> v2:172.21.15.196:3300/0 conn(0x7f74fc0ceff0 0x7f74fc06b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:37.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7502948700 1 --2- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc0b1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:37.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501145700 1 -- 172.21.15.191:0/787853112 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077268873 0 0) 0x7f74fc06c580 con 0x7f74fc0cb8d0 2023-03-24T23:19:37.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501145700 1 -- 172.21.15.191:0/787853112 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e80035b0 con 0x7f74fc0cb8d0 2023-03-24T23:19:37.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501145700 1 -- 172.21.15.191:0/787853112 <== mon.0 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2429817166 0 0) 0x7f74e80035b0 con 0x7f74fc0cb8d0 2023-03-24T23:19:37.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501946700 1 -- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 msgr2=0x7f74fc0b1570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:37.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501946700 1 --2- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc0b1570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501946700 1 -- 172.21.15.191:0/787853112 >> v1:172.21.15.191:6789/0 conn(0x7f74fc0cb8d0 legacy=0x7f74fc160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:37.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501946700 1 -- 172.21.15.191:0/787853112 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74fc06bfc0 con 0x7f74fc0ceff0 2023-03-24T23:19:37.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7502948700 1 --2- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:37.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501946700 1 --2- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0ceff0 0x7f74fc06b6b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009840 tx=0x7f74f8009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34f5da8d1de80d61 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:37.164 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501145700 1 -- 172.21.15.191:0/787853112 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec001670 con 0x7f74fc0ceff0 2023-03-24T23:19:37.164 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501145700 1 -- 172.21.15.191:0/787853112 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f8004030 con 0x7f74fc0ceff0 2023-03-24T23:19:37.164 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.160+0000 7f7501145700 1 -- 172.21.15.191:0/787853112 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f80043a0 con 0x7f74fc0ceff0 2023-03-24T23:19:37.164 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.161+0000 7f75043ab700 1 -- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0ceff0 msgr2=0x7f74fc06b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:37.164 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.161+0000 7f75043ab700 1 --2- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0ceff0 0x7f74fc06b6b0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f74f8009840 tx=0x7f74f8009b50 comp rx=0 tx=0).stop 2023-03-24T23:19:37.164 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.161+0000 7f75043ab700 1 -- 172.21.15.191:0/787853112 shutdown_connections 2023-03-24T23:19:37.164 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.161+0000 7f75043ab700 1 --2- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc0b1570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.165 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.161+0000 7f75043ab700 1 --2- 172.21.15.191:0/787853112 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0ceff0 0x7f74fc06b6b0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.165 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.161+0000 7f75043ab700 1 -- 172.21.15.191:0/787853112 >> 172.21.15.191:0/787853112 conn(0x7f74fc06d880 msgr2=0x7f74fc0cc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:37.165 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.161+0000 7f75043ab700 1 -- 172.21.15.191:0/787853112 shutdown_connections 2023-03-24T23:19:37.165 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.161+0000 7f75043ab700 1 -- 172.21.15.191:0/787853112 wait complete. 2023-03-24T23:19:37.167 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.162+0000 7f75043ab700 1 Processor -- start 2023-03-24T23:19:37.167 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.162+0000 7f75043ab700 1 -- start start 2023-03-24T23:19:37.167 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f75043ab700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc1582e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:37.167 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f75043ab700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0cb8d0 0x7f74fc158820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:37.168 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f75043ab700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74fc0ceff0 0x7f74fc15a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:37.168 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f75043ab700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc1f59d0 con 0x7f74fc0ceff0 2023-03-24T23:19:37.168 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f75043ab700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc1f5cb0 con 0x7f74fc0cb8d0 2023-03-24T23:19:37.168 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f75043ab700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74fc1f5f90 con 0x7f74fc0b10f0 2023-03-24T23:19:37.168 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74fc0ceff0 0x7f74fc15a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:37.168 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74fc0ceff0 0x7f74fc15a030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50020/0 (socket says 172.21.15.191:50020) 2023-03-24T23:19:37.168 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 -- 172.21.15.191:0/2054379685 learned_addr learned my addr 172.21.15.191:0/2054379685 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:37.169 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502147700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc1582e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:37.169 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7501946700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0cb8d0 0x7f74fc158820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:37.169 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 -- 172.21.15.191:0/2054379685 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 msgr2=0x7f74fc1582e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:37.169 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc1582e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.169 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 -- 172.21.15.191:0/2054379685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0cb8d0 msgr2=0x7f74fc158820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:37.169 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0cb8d0 0x7f74fc158820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.169 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 -- 172.21.15.191:0/2054379685 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f4009a70 con 0x7f74fc0ceff0 2023-03-24T23:19:37.170 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.163+0000 7f7502948700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74fc0ceff0 0x7f74fc15a030 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f74f400efc0 tx=0x7f74f400c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:37.170 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.164+0000 7f74f2ffd700 1 -- 172.21.15.191:0/2054379685 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f400ca90 con 0x7f74fc0ceff0 2023-03-24T23:19:37.170 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.164+0000 7f74f2ffd700 1 -- 172.21.15.191:0/2054379685 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74f400f040 con 0x7f74fc0ceff0 2023-03-24T23:19:37.170 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.164+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f80094a0 con 0x7f74fc0ceff0 2023-03-24T23:19:37.170 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.164+0000 7f74f2ffd700 1 -- 172.21.15.191:0/2054379685 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74f400ce30 con 0x7f74fc0ceff0 2023-03-24T23:19:37.170 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.164+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74fc1606b0 con 0x7f74fc0ceff0 2023-03-24T23:19:37.170 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.164+0000 7f74f2ffd700 1 -- 172.21.15.191:0/2054379685 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f74f4003d20 con 0x7f74fc0ceff0 2023-03-24T23:19:37.170 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.164+0000 7f74f2ffd700 1 -- 172.21.15.191:0/2054379685 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f74f4014b50 con 0x7f74fc0ceff0 2023-03-24T23:19:37.171 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.164+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74fc06c2a0 con 0x7f74fc0ceff0 2023-03-24T23:19:37.171 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.167+0000 7f74f2ffd700 1 -- 172.21.15.191:0/2054379685 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f74f4018030 con 0x7f74fc0ceff0 2023-03-24T23:19:37.236 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.236+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74fc073b20 con 0x7f74fc0ceff0 2023-03-24T23:19:37.237 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.237+0000 7f74f2ffd700 1 -- 172.21.15.191:0/2054379685 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f74f4014480 con 0x7f74fc0ceff0 2023-03-24T23:19:37.238 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:37.238 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":10,"fsid":"72b355de-1f4b-4f02-899f-839bc265bc0f","created":"2023-03-24T23:19:34.494902+0000","modified":"2023-03-24T23:19:36.946912+0000","last_up_change":"0.000000","last_in_change":"2023-03-24T23:19:36.946912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0ca3720a-fe6e-4b70-a32f-fef43c5da336","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"55f40987-e709-41aa-84f1-7a0b426e984d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2e378f2-4cdb-4900-8f17-83affc66d8ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b4483b81-cc61-477a-ada6-4620185b9f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f133b7f6-ca18-4ff3-a71f-d9ae1593ce57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b284d515-2796-4d33-81a8-0e42195d49bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37ee78d6-7798-4d17-ad11-4fffd8355d73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0c1c2708-0d59-4f5b-89aa-75ea67df828e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-24T23:19:37.239 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74fc0ceff0 msgr2=0x7f74fc15a030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:37.239 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74fc0ceff0 0x7f74fc15a030 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f74f400efc0 tx=0x7f74f400c3a0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.239 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 shutdown_connections 2023-03-24T23:19:37.239 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f74fc0ceff0 0x7f74fc15a030 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.239 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f74fc0b10f0 0x7f74fc1582e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.239 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 --2- 172.21.15.191:0/2054379685 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f74fc0cb8d0 0x7f74fc158820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:37.240 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 >> 172.21.15.191:0/2054379685 conn(0x7f74fc06d880 msgr2=0x7f74fc15fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:37.240 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 shutdown_connections 2023-03-24T23:19:37.240 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:37.238+0000 7f75043ab700 1 -- 172.21.15.191:0/2054379685 wait complete. 2023-03-24T23:19:37.250 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-03-24T23:19:37.251 INFO:tasks.ceph:Waiting for OSDs to come up 2023-03-24T23:19:37.555 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:37.555+0000 7f6bb5d9b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-24T23:19:37.590 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-03-24T23:19:37.590+0000 7fe15094e640 -1 Falling back to public interface 2023-03-24T23:19:37.597 INFO:tasks.ceph.osd.2.smithi191.stderr:2023-03-24T23:19:37.597+0000 7f5f2486e640 -1 Falling back to public interface 2023-03-24T23:19:37.598 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:37.598+0000 7fc54dc46200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-03-24T23:19:37.600 INFO:tasks.ceph.osd.5.smithi196.stderr:2023-03-24T23:19:37.600+0000 7f216afa5640 -1 Falling back to public interface 2023-03-24T23:19:37.604 INFO:tasks.ceph.osd.4.smithi196.stderr:2023-03-24T23:19:37.603+0000 7f6917d52640 -1 Falling back to public interface 2023-03-24T23:19:37.606 INFO:tasks.ceph.osd.0.smithi191.stderr:2023-03-24T23:19:37.605+0000 7fea31b52640 -1 Falling back to public interface 2023-03-24T23:19:37.606 INFO:tasks.ceph.osd.1.smithi191.stderr:2023-03-24T23:19:37.606+0000 7f738a2e3640 -1 Falling back to public interface 2023-03-24T23:19:37.607 INFO:tasks.ceph.osd.0.smithi191.stderr:2023-03-24T23:19:37.606+0000 7fea28f10700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2023-03-24T23:19:37.614 INFO:tasks.ceph.osd.6.smithi196.stderr:2023-03-24T23:19:37.614+0000 7f2cbcdc1640 -1 Falling back to public interface 2023-03-24T23:19:37.618 INFO:tasks.ceph.osd.7.smithi196.stderr:2023-03-24T23:19:37.618+0000 7f565df8e640 -1 Falling back to public interface 2023-03-24T23:19:37.636 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:37.636+0000 7f6bb5d9b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-24T23:19:37.682 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:37.681+0000 7fc54dc46200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-03-24T23:19:37.775 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:37.774+0000 7f6bb5d9b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-24T23:19:37.821 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:37.821+0000 7fc54dc46200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-03-24T23:19:38.005 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:38.005+0000 7f6bb5d9b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-24T23:19:38.054 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:38.053+0000 7fc54dc46200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-03-24T23:19:38.057 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-24T23:19:38.079 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:38.079+0000 7f6bb5d9b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-24T23:19:38.127 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:38.126+0000 7fc54dc46200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-03-24T23:19:38.170 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.170+0000 7f524eecd700 1 Processor -- start 2023-03-24T23:19:38.171 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.170+0000 7f524eecd700 1 -- start start 2023-03-24T23:19:38.172 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524eecd700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52481571c0 0x7f52480cfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:38.172 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524eecd700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f52480cdc40 0x7f52480ce090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:38.172 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524eecd700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5248157590 con 0x7f52480cc980 2023-03-24T23:19:38.173 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524eecd700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524815e140 con 0x7f52481571c0 2023-03-24T23:19:38.173 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524eecd700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f524815d530 con 0x7f52480cdc40 2023-03-24T23:19:38.173 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524cc69700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52481571c0 0x7f52480cfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:38.173 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524cc69700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f52481571c0 0x7f52480cfef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50022/0 (socket says 172.21.15.191:50022) 2023-03-24T23:19:38.173 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524cc69700 1 -- 172.21.15.191:0/251504564 learned_addr learned my addr 172.21.15.191:0/251504564 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:38.173 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f5247fff700 1 --2- 172.21.15.191:0/251504564 >> v2:172.21.15.196:3300/0 conn(0x7f52480cdc40 0x7f52480ce090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:38.173 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524cc69700 1 -- 172.21.15.191:0/251504564 >> v2:172.21.15.196:3300/0 conn(0x7f52480cdc40 msgr2=0x7f52480ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:38.174 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524cc69700 1 --2- 172.21.15.191:0/251504564 >> v2:172.21.15.196:3300/0 conn(0x7f52480cdc40 0x7f52480ce090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.174 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524cc69700 1 -- 172.21.15.191:0/251504564 >> v1:172.21.15.196:6789/0 conn(0x7f52480cc980 legacy=0x7f5248160ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:38.174 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.171+0000 7f524cc69700 1 -- 172.21.15.191:0/251504564 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f524815d810 con 0x7f52481571c0 2023-03-24T23:19:38.174 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f524cc69700 1 --2- 172.21.15.191:0/251504564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52481571c0 0x7f52480cfef0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f52480b6a20 tx=0x7f523c009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a851bd347b8a916 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:38.174 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f52477fe700 1 -- 172.21.15.191:0/251504564 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c011830 con 0x7f52481571c0 2023-03-24T23:19:38.174 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f52477fe700 1 -- 172.21.15.191:0/251504564 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c011990 con 0x7f52481571c0 2023-03-24T23:19:38.174 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f52477fe700 1 -- 172.21.15.191:0/251504564 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c011d20 con 0x7f52481571c0 2023-03-24T23:19:38.175 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f524eecd700 1 -- 172.21.15.191:0/251504564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52481571c0 msgr2=0x7f52480cfef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:38.175 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f524eecd700 1 --2- 172.21.15.191:0/251504564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52481571c0 0x7f52480cfef0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f52480b6a20 tx=0x7f523c009bd0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.176 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f524eecd700 1 -- 172.21.15.191:0/251504564 shutdown_connections 2023-03-24T23:19:38.176 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f524eecd700 1 --2- 172.21.15.191:0/251504564 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52481571c0 0x7f52480cfef0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.176 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f524eecd700 1 --2- 172.21.15.191:0/251504564 >> v2:172.21.15.196:3300/0 conn(0x7f52480cdc40 0x7f52480ce090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.176 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.172+0000 7f524eecd700 1 -- 172.21.15.191:0/251504564 >> 172.21.15.191:0/251504564 conn(0x7f5248070270 msgr2=0x7f5248070640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:38.176 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.173+0000 7f524eecd700 1 -- 172.21.15.191:0/251504564 shutdown_connections 2023-03-24T23:19:38.176 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.173+0000 7f524eecd700 1 -- 172.21.15.191:0/251504564 wait complete. 2023-03-24T23:19:38.176 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.174+0000 7f524eecd700 1 Processor -- start 2023-03-24T23:19:38.176 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.174+0000 7f524eecd700 1 -- start start 2023-03-24T23:19:38.178 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.175+0000 7f524eecd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52480cc980 0x7f5248152000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:38.178 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.175+0000 7f524eecd700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52480cdc40 0x7f52481f1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:38.178 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524eecd700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52481571c0 0x7f524813f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:38.179 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524eecd700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52481412b0 con 0x7f52480cc980 2023-03-24T23:19:38.179 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524eecd700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5248152540 con 0x7f52480cdc40 2023-03-24T23:19:38.179 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524eecd700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52481f2460 con 0x7f52481571c0 2023-03-24T23:19:38.179 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52480cc980 0x7f5248152000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:38.179 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52480cc980 0x7f5248152000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50028/0 (socket says 172.21.15.191:50028) 2023-03-24T23:19:38.179 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 -- 172.21.15.191:0/723089408 learned_addr learned my addr 172.21.15.191:0/723089408 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:38.179 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f5247fff700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52480cdc40 0x7f52481f1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:38.180 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524d46a700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52481571c0 0x7f524813f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:38.180 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 -- 172.21.15.191:0/723089408 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52481571c0 msgr2=0x7f524813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:38.180 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52481571c0 0x7f524813f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.180 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 -- 172.21.15.191:0/723089408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52480cdc40 msgr2=0x7f52481f1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:38.180 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52480cdc40 0x7f52481f1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.180 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 -- 172.21.15.191:0/723089408 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c009550 con 0x7f52480cc980 2023-03-24T23:19:38.180 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524cc69700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52480cc980 0x7f5248152000 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f523c000c00 tx=0x7f523c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:38.180 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f52457fa700 1 -- 172.21.15.191:0/723089408 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c004380 con 0x7f52480cc980 2023-03-24T23:19:38.181 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f52457fa700 1 -- 172.21.15.191:0/723089408 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f523c01f070 con 0x7f52480cc980 2023-03-24T23:19:38.181 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f524815ddd0 con 0x7f52480cc980 2023-03-24T23:19:38.181 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f52457fa700 1 -- 172.21.15.191:0/723089408 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f523c0036a0 con 0x7f52480cc980 2023-03-24T23:19:38.181 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.176+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f524815daf0 con 0x7f52480cc980 2023-03-24T23:19:38.181 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.177+0000 7f52457fa700 1 -- 172.21.15.191:0/723089408 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f523c021030 con 0x7f52480cc980 2023-03-24T23:19:38.181 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.177+0000 7f52457fa700 1 -- 172.21.15.191:0/723089408 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f523c02d440 con 0x7f52480cc980 2023-03-24T23:19:38.181 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.177+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f524813e9a0 con 0x7f52480cc980 2023-03-24T23:19:38.182 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.179+0000 7f52457fa700 1 -- 172.21.15.191:0/723089408 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72437 (secure 0 0 0) 0x7f523c015020 con 0x7f52480cc980 2023-03-24T23:19:38.249 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.249+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52480735d0 con 0x7f52480cc980 2023-03-24T23:19:38.249 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.250+0000 7f52457fa700 1 -- 172.21.15.191:0/723089408 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f523c003d00 con 0x7f52480cc980 2023-03-24T23:19:38.250 INFO:teuthology.misc.health.smithi191.stdout: 2023-03-24T23:19:38.250 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":10,"fsid":"72b355de-1f4b-4f02-899f-839bc265bc0f","created":"2023-03-24T23:19:34.494902+0000","modified":"2023-03-24T23:19:36.946912+0000","last_up_change":"0.000000","last_in_change":"2023-03-24T23:19:36.946912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0ca3720a-fe6e-4b70-a32f-fef43c5da336","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"55f40987-e709-41aa-84f1-7a0b426e984d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2e378f2-4cdb-4900-8f17-83affc66d8ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b4483b81-cc61-477a-ada6-4620185b9f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f133b7f6-ca18-4ff3-a71f-d9ae1593ce57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b284d515-2796-4d33-81a8-0e42195d49bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37ee78d6-7798-4d17-ad11-4fffd8355d73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0c1c2708-0d59-4f5b-89aa-75ea67df828e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-24T23:19:38.251 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.250+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52480cc980 msgr2=0x7f5248152000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:38.251 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.250+0000 7f524eecd700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52480cc980 0x7f5248152000 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f523c000c00 tx=0x7f523c004750 comp rx=0 tx=0).stop 2023-03-24T23:19:38.251 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.250+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 shutdown_connections 2023-03-24T23:19:38.252 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.250+0000 7f524eecd700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f52480cc980 0x7f5248152000 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.252 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.250+0000 7f524eecd700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52481571c0 0x7f524813f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.252 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.250+0000 7f524eecd700 1 --2- 172.21.15.191:0/723089408 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52480cdc40 0x7f52481f1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:38.252 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.250+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 >> 172.21.15.191:0/723089408 conn(0x7f5248070270 msgr2=0x7f52481567b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:38.252 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.251+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 shutdown_connections 2023-03-24T23:19:38.252 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:38.251+0000 7f524eecd700 1 -- 172.21.15.191:0/723089408 wait complete. 2023-03-24T23:19:38.261 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-24T23:19:38.363 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:38.363+0000 7f6bb5d9b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-24T23:19:38.403 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:38.402+0000 7fc54dc46200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-03-24T23:19:38.580 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:38.580+0000 7f6bb5d9b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-24T23:19:38.623 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:38.622+0000 7fc54dc46200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-03-24T23:19:38.734 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:38.734+0000 7f6bb5d9b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-24T23:19:38.780 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:38.779+0000 7fc54dc46200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-03-24T23:19:38.848 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:38.848+0000 7f6bb5d9b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-24T23:19:38.895 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:38.894+0000 7fc54dc46200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-03-24T23:19:38.994 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:38.994+0000 7f6bb5d9b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-24T23:19:39.042 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:39.042+0000 7fc54dc46200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-03-24T23:19:39.271 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:39.271+0000 7f6bb5d9b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-24T23:19:39.319 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:39.319+0000 7fc54dc46200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-03-24T23:19:39.455 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:39.455+0000 7f6bb5d9b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-24T23:19:39.502 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:39.502+0000 7fc54dc46200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-03-24T23:19:40.039 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:40.039+0000 7f6bb5d9b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-24T23:19:40.089 INFO:tasks.ceph.mgr.x.smithi196.stderr:2023-03-24T23:19:40.089+0000 7fc54dc46200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-03-24T23:19:41.064 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:41.064+0000 7f6b6e040700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-03-24T23:19:41.064 INFO:tasks.ceph.mgr.y.smithi191.stderr:2023-03-24T23:19:41.064+0000 7f6b6e040700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-03-24T23:19:41.137 INFO:tasks.ceph.osd.6.smithi196.stderr:2023-03-24T23:19:41.137+0000 7f2cbcdc1640 -1 osd.6 0 log_to_monitors true 2023-03-24T23:19:41.143 INFO:tasks.ceph.osd.5.smithi196.stderr:2023-03-24T23:19:41.143+0000 7f216afa5640 -1 osd.5 0 log_to_monitors true 2023-03-24T23:19:41.148 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-03-24T23:19:41.148+0000 7fe15094e640 -1 osd.3 0 log_to_monitors true 2023-03-24T23:19:41.168 INFO:tasks.ceph.osd.7.smithi196.stderr:2023-03-24T23:19:41.168+0000 7f565df8e640 -1 osd.7 0 log_to_monitors true 2023-03-24T23:19:41.168 INFO:tasks.ceph.osd.4.smithi196.stderr:2023-03-24T23:19:41.168+0000 7f6917d52640 -1 osd.4 0 log_to_monitors true 2023-03-24T23:19:41.176 INFO:tasks.ceph.osd.0.smithi191.stderr:2023-03-24T23:19:41.176+0000 7fea31b52640 -1 osd.0 0 log_to_monitors true 2023-03-24T23:19:41.191 INFO:tasks.ceph.osd.2.smithi191.stderr:2023-03-24T23:19:41.191+0000 7f5f2486e640 -1 osd.2 0 log_to_monitors true 2023-03-24T23:19:41.200 INFO:tasks.ceph.osd.1.smithi191.stderr:2023-03-24T23:19:41.200+0000 7f738a2e3640 -1 osd.1 0 log_to_monitors true 2023-03-24T23:19:44.928 INFO:tasks.ceph.osd.7.smithi196.stderr:2023-03-24T23:19:44.925+0000 7f5652346700 -1 osd.7 0 waiting for initial osdmap 2023-03-24T23:19:44.946 INFO:tasks.ceph.osd.7.smithi196.stderr:2023-03-24T23:19:44.946+0000 7f5649f20700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-24T23:19:45.063 INFO:tasks.ceph.osd.4.smithi196.stderr:2023-03-24T23:19:45.063+0000 7f690c10a700 -1 osd.4 0 waiting for initial osdmap 2023-03-24T23:19:45.066 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-24T23:19:45.068 INFO:tasks.ceph.osd.1.smithi191.stderr:2023-03-24T23:19:45.067+0000 7f737e69b700 -1 osd.1 0 waiting for initial osdmap 2023-03-24T23:19:45.074 INFO:tasks.ceph.osd.5.smithi196.stderr:2023-03-24T23:19:45.074+0000 7f215f35d700 -1 osd.5 0 waiting for initial osdmap 2023-03-24T23:19:45.078 INFO:tasks.ceph.osd.4.smithi196.stderr:2023-03-24T23:19:45.079+0000 7f6903ce4700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-24T23:19:45.086 INFO:tasks.ceph.osd.5.smithi196.stderr:2023-03-24T23:19:45.087+0000 7f2156f37700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-24T23:19:45.090 INFO:tasks.ceph.osd.2.smithi191.stderr:2023-03-24T23:19:45.090+0000 7f5f18c26700 -1 osd.2 0 waiting for initial osdmap 2023-03-24T23:19:45.102 INFO:tasks.ceph.osd.6.smithi196.stderr:2023-03-24T23:19:45.102+0000 7f2cb1179700 -1 osd.6 0 waiting for initial osdmap 2023-03-24T23:19:45.108 INFO:tasks.ceph.osd.1.smithi191.stderr:2023-03-24T23:19:45.108+0000 7f7376275700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-24T23:19:45.112 INFO:tasks.ceph.osd.0.smithi191.stderr:2023-03-24T23:19:45.112+0000 7fea25f0a700 -1 osd.0 0 waiting for initial osdmap 2023-03-24T23:19:45.115 INFO:tasks.ceph.osd.6.smithi196.stderr:2023-03-24T23:19:45.116+0000 7f2ca8d53700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-24T23:19:45.117 INFO:tasks.ceph.osd.2.smithi191.stderr:2023-03-24T23:19:45.117+0000 7f5f10800700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-24T23:19:45.120 INFO:tasks.ceph.osd.0.smithi191.stderr:2023-03-24T23:19:45.120+0000 7fea1dae4700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-24T23:19:45.121 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-03-24T23:19:45.120+0000 7fe144d06700 -1 osd.3 0 waiting for initial osdmap 2023-03-24T23:19:45.137 INFO:tasks.ceph.osd.3.smithi191.stderr:2023-03-24T23:19:45.137+0000 7fe13c8e0700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-03-24T23:19:45.201 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.200+0000 7f528450e700 1 Processor -- start 2023-03-24T23:19:45.202 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.201+0000 7f528450e700 1 -- start start 2023-03-24T23:19:45.204 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.201+0000 7f528450e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c15c8b0 0x7f527c0cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:45.205 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.201+0000 7f528450e700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f527c0b1800 0x7f527c0cb5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:45.205 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.201+0000 7f528450e700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f527c0cbbe0 con 0x7f527c161270 2023-03-24T23:19:45.205 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.201+0000 7f528450e700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f527c0cbec0 con 0x7f527c0b1800 2023-03-24T23:19:45.205 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f528450e700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c160790 con 0x7f527c15c8b0 2023-03-24T23:19:45.205 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f52822aa700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c15c8b0 0x7f527c0cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:45.206 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f5282aab700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f527c0b1800 0x7f527c0cb5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:45.206 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f5281aa9700 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f527c161270 0x7f527c0b10f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.191:50620/0 (socket says 172.21.15.191:50620) 2023-03-24T23:19:45.206 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f5281aa9700 1 -- 172.21.15.191:0/1652756398 learned_addr learned my addr 172.21.15.191:0/1652756398 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:45.206 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f52822aa700 1 -- 172.21.15.191:0/1652756398 >> v2:172.21.15.196:3300/0 conn(0x7f527c0b1800 msgr2=0x7f527c0cb5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:45.206 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f52822aa700 1 --2- 172.21.15.191:0/1652756398 >> v2:172.21.15.196:3300/0 conn(0x7f527c0b1800 0x7f527c0cb5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.207 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f52822aa700 1 -- 172.21.15.191:0/1652756398 >> v1:172.21.15.196:6789/0 conn(0x7f527c161270 legacy=0x7f527c0b10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:45.207 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f52822aa700 1 -- 172.21.15.191:0/1652756398 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f527c160a70 con 0x7f527c15c8b0 2023-03-24T23:19:45.207 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f52812a8700 1 -- 172.21.15.191:0/1652756398 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248641317 0 0) 0x7f527c0cbbe0 con 0x7f527c161270 2023-03-24T23:19:45.207 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f5282aab700 1 --2- 172.21.15.191:0/1652756398 >> v2:172.21.15.196:3300/0 conn(0x7f527c0b1800 0x7f527c0cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-03-24T23:19:45.207 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.202+0000 7f52822aa700 1 --2- 172.21.15.191:0/1652756398 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c15c8b0 0x7f527c0cb090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5278009840 tx=0x7f5278009b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=849c6f9462535059 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:45.208 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f52812a8700 1 -- 172.21.15.191:0/1652756398 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278011830 con 0x7f527c15c8b0 2023-03-24T23:19:45.208 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f52812a8700 1 -- 172.21.15.191:0/1652756398 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278011990 con 0x7f527c15c8b0 2023-03-24T23:19:45.208 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f52812a8700 1 -- 172.21.15.191:0/1652756398 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5278011d20 con 0x7f527c15c8b0 2023-03-24T23:19:45.208 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f528450e700 1 -- 172.21.15.191:0/1652756398 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c15c8b0 msgr2=0x7f527c0cb090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:45.208 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f528450e700 1 --2- 172.21.15.191:0/1652756398 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c15c8b0 0x7f527c0cb090 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5278009840 tx=0x7f5278009b50 comp rx=0 tx=0).stop 2023-03-24T23:19:45.209 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f528450e700 1 -- 172.21.15.191:0/1652756398 shutdown_connections 2023-03-24T23:19:45.209 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f528450e700 1 --2- 172.21.15.191:0/1652756398 >> v2:172.21.15.196:3300/0 conn(0x7f527c0b1800 0x7f527c0cb5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.209 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f528450e700 1 --2- 172.21.15.191:0/1652756398 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c15c8b0 0x7f527c0cb090 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.209 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f528450e700 1 -- 172.21.15.191:0/1652756398 >> 172.21.15.191:0/1652756398 conn(0x7f527c06d880 msgr2=0x7f527c0cc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:45.209 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f528450e700 1 -- 172.21.15.191:0/1652756398 shutdown_connections 2023-03-24T23:19:45.209 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.203+0000 7f528450e700 1 -- 172.21.15.191:0/1652756398 wait complete. 2023-03-24T23:19:45.210 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.204+0000 7f528450e700 1 Processor -- start 2023-03-24T23:19:45.210 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.204+0000 7f528450e700 1 -- start start 2023-03-24T23:19:45.213 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f528450e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f527c0b1800 0x7f527c150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:45.213 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f528450e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527c15c8b0 0x7f527c0af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:45.213 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f528450e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c161270 0x7f527c0af7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:45.213 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f528450e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c153df0 con 0x7f527c15c8b0 2023-03-24T23:19:45.213 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f528450e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f527c151310 con 0x7f527c0b1800 2023-03-24T23:19:45.214 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f528450e700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f527c0c33d0 con 0x7f527c161270 2023-03-24T23:19:45.214 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f5281aa9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527c15c8b0 0x7f527c0af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:45.214 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f5281aa9700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527c15c8b0 0x7f527c0af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50140/0 (socket says 172.21.15.191:50140) 2023-03-24T23:19:45.214 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f5281aa9700 1 -- 172.21.15.191:0/1761462030 learned_addr learned my addr 172.21.15.191:0/1761462030 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:45.214 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f52822aa700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f527c0b1800 0x7f527c150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:45.214 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.205+0000 7f5282aab700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c161270 0x7f527c0af7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:45.214 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f5281aa9700 1 -- 172.21.15.191:0/1761462030 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c161270 msgr2=0x7f527c0af7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:45.215 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f5281aa9700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c161270 0x7f527c0af7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.215 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f5281aa9700 1 -- 172.21.15.191:0/1761462030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f527c0b1800 msgr2=0x7f527c150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:45.215 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f5281aa9700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f527c0b1800 0x7f527c150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.215 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f5281aa9700 1 -- 172.21.15.191:0/1761462030 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5274009a70 con 0x7f527c15c8b0 2023-03-24T23:19:45.215 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f5281aa9700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527c15c8b0 0x7f527c0af270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f527000c520 tx=0x7f527000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:45.215 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f526effd700 1 -- 172.21.15.191:0/1761462030 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5270004c80 con 0x7f527c15c8b0 2023-03-24T23:19:45.215 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f526effd700 1 -- 172.21.15.191:0/1761462030 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5270004de0 con 0x7f527c15c8b0 2023-03-24T23:19:45.215 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52780094a0 con 0x7f527c15c8b0 2023-03-24T23:19:45.216 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f526effd700 1 -- 172.21.15.191:0/1761462030 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52700072d0 con 0x7f527c15c8b0 2023-03-24T23:19:45.216 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.206+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f527c072a50 con 0x7f527c15c8b0 2023-03-24T23:19:45.216 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.207+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f527c160d50 con 0x7f527c15c8b0 2023-03-24T23:19:45.216 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.207+0000 7f526effd700 1 -- 172.21.15.191:0/1761462030 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 4) v1 ==== 87553+0+0 (secure 0 0 0) 0x7f5270007430 con 0x7f527c15c8b0 2023-03-24T23:19:45.216 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.208+0000 7f526effd700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f5260069d50 0x7f526006c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:45.216 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.208+0000 7f52822aa700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f5260069d50 0x7f526006c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:45.216 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.208+0000 7f526effd700 1 -- 172.21.15.191:0/1761462030 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f5270086c50 con 0x7f527c15c8b0 2023-03-24T23:19:45.217 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.208+0000 7f52822aa700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f5260069d50 0x7f526006c200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5278009fd0 tx=0x7f5278008030 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:45.217 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.212+0000 7f526effd700 1 -- 172.21.15.191:0/1761462030 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f52700533b0 con 0x7f527c15c8b0 2023-03-24T23:19:45.400 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.400+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f527c15cc80 con 0x7f527c15c8b0 2023-03-24T23:19:45.401 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.400+0000 7f526effd700 1 -- 172.21.15.191:0/1761462030 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f52700569d0 con 0x7f527c15c8b0 2023-03-24T23:19:45.401 INFO:teuthology.misc.health.smithi191.stdout: 2023-03-24T23:19:45.401 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":12,"fsid":"72b355de-1f4b-4f02-899f-839bc265bc0f","created":"2023-03-24T23:19:34.494902+0000","modified":"2023-03-24T23:19:43.060933+0000","last_up_change":"0.000000","last_in_change":"2023-03-24T23:19:36.946912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"0ca3720a-fe6e-4b70-a32f-fef43c5da336","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"55f40987-e709-41aa-84f1-7a0b426e984d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b2e378f2-4cdb-4900-8f17-83affc66d8ce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b4483b81-cc61-477a-ada6-4620185b9f6b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f133b7f6-ca18-4ff3-a71f-d9ae1593ce57","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b284d515-2796-4d33-81a8-0e42195d49bb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"37ee78d6-7798-4d17-ad11-4fffd8355d73","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0c1c2708-0d59-4f5b-89aa-75ea67df828e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-24T23:19:45.403 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f5260069d50 msgr2=0x7f526006c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:45.403 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f5260069d50 0x7f526006c200 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5278009fd0 tx=0x7f5278008030 comp rx=0 tx=0).stop 2023-03-24T23:19:45.403 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527c15c8b0 msgr2=0x7f527c0af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:45.403 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527c15c8b0 0x7f527c0af270 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f527000c520 tx=0x7f527000c830 comp rx=0 tx=0).stop 2023-03-24T23:19:45.403 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 shutdown_connections 2023-03-24T23:19:45.404 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f5260069d50 0x7f526006c200 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.404 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f527c161270 0x7f527c0af7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.404 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f527c0b1800 0x7f527c150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.404 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 --2- 172.21.15.191:0/1761462030 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f527c15c8b0 0x7f527c0af270 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:45.404 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.402+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 >> 172.21.15.191:0/1761462030 conn(0x7f527c06d880 msgr2=0x7f527c0b0d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:45.404 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.403+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 shutdown_connections 2023-03-24T23:19:45.404 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:45.403+0000 7f528450e700 1 -- 172.21.15.191:0/1761462030 wait complete. 2023-03-24T23:19:45.414 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-03-24T23:19:52.219 DEBUG:teuthology.orchestra.run.smithi191:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-03-24T23:19:52.343 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.342+0000 7f7dc816f700 1 Processor -- start 2023-03-24T23:19:52.343 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.343+0000 7f7dc816f700 1 -- start start 2023-03-24T23:19:52.344 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.343+0000 7f7dc816f700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7dc006f4d0 0x7f7dc00cb590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.345 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.343+0000 7f7dc816f700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc015cd60 con 0x7f7dc015c990 2023-03-24T23:19:52.345 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.343+0000 7f7dc816f700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc00ca430 con 0x7f7dc006f030 2023-03-24T23:19:52.345 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.343+0000 7f7dc816f700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc00b3bb0 con 0x7f7dc006f4d0 2023-03-24T23:19:52.345 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc5f0b700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f7dc015c990 0x7f7dc0070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57770/0 (socket says 172.21.15.191:57770) 2023-03-24T23:19:52.345 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc5f0b700 1 -- 172.21.15.191:0/2348166439 learned_addr learned my addr 172.21.15.191:0/2348166439 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:52.345 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc670c700 1 --2- 172.21.15.191:0/2348166439 >> v2:172.21.15.196:3300/0 conn(0x7f7dc006f4d0 0x7f7dc00cb590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.346 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc4f09700 1 -- 172.21.15.191:0/2348166439 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105649613 0 0) 0x7f7db0001110 con 0x7f7dc015c990 2023-03-24T23:19:52.346 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc4f09700 1 -- 172.21.15.191:0/2348166439 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dac0035b0 con 0x7f7dc015c990 2023-03-24T23:19:52.346 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc4f09700 1 -- 172.21.15.191:0/2348166439 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834443005 0 0) 0x7f7dc00ca430 con 0x7f7dc006f030 2023-03-24T23:19:52.346 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc4f09700 1 -- 172.21.15.191:0/2348166439 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7db0001110 con 0x7f7dc006f030 2023-03-24T23:19:52.346 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc670c700 1 -- 172.21.15.191:0/2348166439 >> v1:172.21.15.196:6789/0 conn(0x7f7dc006f030 legacy=0x7f7dc00cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.346 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc670c700 1 -- 172.21.15.191:0/2348166439 >> v1:172.21.15.191:6789/0 conn(0x7f7dc015c990 legacy=0x7f7dc0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.346 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc670c700 1 -- 172.21.15.191:0/2348166439 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc00719a0 con 0x7f7dc006f4d0 2023-03-24T23:19:52.346 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc670c700 1 --2- 172.21.15.191:0/2348166439 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 0x7f7dc00cb590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009d90 tx=0x7f7db800b330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49ea7bcc5b22c108 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:52.347 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc4f09700 1 -- 172.21.15.191:0/2348166439 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db800e070 con 0x7f7dc006f4d0 2023-03-24T23:19:52.347 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc4f09700 1 -- 172.21.15.191:0/2348166439 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db800bb20 con 0x7f7dc006f4d0 2023-03-24T23:19:52.347 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.344+0000 7f7dc4f09700 1 -- 172.21.15.191:0/2348166439 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db800be90 con 0x7f7dc006f4d0 2023-03-24T23:19:52.349 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.345+0000 7f7dc816f700 1 -- 172.21.15.191:0/2348166439 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 msgr2=0x7f7dc00cb590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.349 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.345+0000 7f7dc816f700 1 --2- 172.21.15.191:0/2348166439 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 0x7f7dc00cb590 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f7db8009d90 tx=0x7f7db800b330 comp rx=0 tx=0).stop 2023-03-24T23:19:52.349 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.345+0000 7f7dc816f700 1 -- 172.21.15.191:0/2348166439 shutdown_connections 2023-03-24T23:19:52.350 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.345+0000 7f7dc816f700 1 --2- 172.21.15.191:0/2348166439 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 0x7f7dc00cb590 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.350 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.345+0000 7f7dc816f700 1 -- 172.21.15.191:0/2348166439 >> 172.21.15.191:0/2348166439 conn(0x7f7dc00cc8f0 msgr2=0x7f7dc00cccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:52.350 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.345+0000 7f7dc816f700 1 -- 172.21.15.191:0/2348166439 shutdown_connections 2023-03-24T23:19:52.350 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.345+0000 7f7dc816f700 1 -- 172.21.15.191:0/2348166439 wait complete. 2023-03-24T23:19:52.350 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.346+0000 7f7dc816f700 1 Processor -- start 2023-03-24T23:19:52.350 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.346+0000 7f7dc816f700 1 -- start start 2023-03-24T23:19:52.350 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc816f700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc006f030 0x7f7dc015fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.351 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc816f700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 0x7f7dc0146e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.351 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc816f700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7dc015c990 0x7f7dc00c70e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.351 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc816f700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc015f820 con 0x7f7dc006f030 2023-03-24T23:19:52.351 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc816f700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc0147370 con 0x7f7dc006f4d0 2023-03-24T23:19:52.351 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc816f700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc00ca280 con 0x7f7dc015c990 2023-03-24T23:19:52.351 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc5f0b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc006f030 0x7f7dc015fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.351 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc5f0b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc006f030 0x7f7dc015fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50374/0 (socket says 172.21.15.191:50374) 2023-03-24T23:19:52.352 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc5f0b700 1 -- 172.21.15.191:0/374793677 learned_addr learned my addr 172.21.15.191:0/374793677 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:52.352 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc670c700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7dc015c990 0x7f7dc00c70e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.352 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc570a700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 0x7f7dc0146e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.352 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.347+0000 7f7dc5f0b700 1 -- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7dc015c990 msgr2=0x7f7dc00c70e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.352 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc5f0b700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7dc015c990 0x7f7dc00c70e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.352 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc5f0b700 1 -- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 msgr2=0x7f7dc0146e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.352 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc5f0b700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 0x7f7dc0146e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.352 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc5f0b700 1 -- 172.21.15.191:0/374793677 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db8009a70 con 0x7f7dc006f030 2023-03-24T23:19:52.353 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc570a700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 0x7f7dc0146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:52.353 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc670c700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7dc015c990 0x7f7dc00c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:52.353 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc5f0b700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc006f030 0x7f7dc015fc40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7db000c700 tx=0x7f7db000cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:52.353 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7db67fc700 1 -- 172.21.15.191:0/374793677 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0005d70 con 0x7f7dc006f030 2023-03-24T23:19:52.353 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7db67fc700 1 -- 172.21.15.191:0/374793677 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db00070f0 con 0x7f7dc006f030 2023-03-24T23:19:52.353 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7db67fc700 1 -- 172.21.15.191:0/374793677 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db00074a0 con 0x7f7dc006f030 2023-03-24T23:19:52.354 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc00ca680 con 0x7f7dc006f030 2023-03-24T23:19:52.354 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.348+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc0071c80 con 0x7f7dc006f030 2023-03-24T23:19:52.354 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.349+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc00ca960 con 0x7f7dc006f030 2023-03-24T23:19:52.354 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.349+0000 7f7db67fc700 1 -- 172.21.15.191:0/374793677 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f7db0007600 con 0x7f7dc006f030 2023-03-24T23:19:52.355 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.350+0000 7f7db67fc700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f7da4069de0 0x7f7da406c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.355 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.350+0000 7f7dc570a700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f7da4069de0 0x7f7da406c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.355 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.350+0000 7f7db67fc700 1 -- 172.21.15.191:0/374793677 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f7db0087030 con 0x7f7dc006f030 2023-03-24T23:19:52.355 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.350+0000 7f7dc570a700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f7da4069de0 0x7f7da406c290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0160400 tx=0x7f7dbc009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:52.356 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.354+0000 7f7db67fc700 1 -- 172.21.15.191:0/374793677 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f7db0052dc0 con 0x7f7dc006f030 2023-03-24T23:19:52.545 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.545+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dc015cd60 con 0x7f7dc006f030 2023-03-24T23:19:52.546 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.545+0000 7f7db67fc700 1 -- 172.21.15.191:0/374793677 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12430 (secure 0 0 0) 0x7f7dc015cd60 con 0x7f7dc006f030 2023-03-24T23:19:52.546 INFO:teuthology.misc.health.smithi191.stdout: 2023-03-24T23:19:52.546 INFO:teuthology.misc.health.smithi191.stdout:{"epoch":16,"fsid":"72b355de-1f4b-4f02-899f-839bc265bc0f","created":"2023-03-24T23:19:34.494902+0000","modified":"2023-03-24T23:19:48.958468+0000","last_up_change":"2023-03-24T23:19:45.928145+0000","last_in_change":"2023-03-24T23:19:36.946912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-24T23:19:46.103710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ca3720a-fe6e-4b70-a32f-fef43c5da336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6817","nonce":97650},{"type":"v1","addr":"172.21.15.191:6819","nonce":97650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6821","nonce":97650},{"type":"v1","addr":"172.21.15.191:6823","nonce":97650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6829","nonce":97650},{"type":"v1","addr":"172.21.15.191:6831","nonce":97650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":97650},{"type":"v1","addr":"172.21.15.191:6827","nonce":97650}]},"public_addr":"172.21.15.191:6819/97650","cluster_addr":"172.21.15.191:6823/97650","heartbeat_back_addr":"172.21.15.191:6831/97650","heartbeat_front_addr":"172.21.15.191:6827/97650","state":["exists","up"]},{"osd":1,"uuid":"55f40987-e709-41aa-84f1-7a0b426e984d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":97651},{"type":"v1","addr":"172.21.15.191:6818","nonce":97651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":97651},{"type":"v1","addr":"172.21.15.191:6822","nonce":97651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":97651},{"type":"v1","addr":"172.21.15.191:6830","nonce":97651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":97651},{"type":"v1","addr":"172.21.15.191:6826","nonce":97651}]},"public_addr":"172.21.15.191:6818/97651","cluster_addr":"172.21.15.191:6822/97651","heartbeat_back_addr":"172.21.15.191:6830/97651","heartbeat_front_addr":"172.21.15.191:6826/97651","state":["exists","up"]},{"osd":2,"uuid":"b2e378f2-4cdb-4900-8f17-83affc66d8ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":97648},{"type":"v1","addr":"172.21.15.191:6809","nonce":97648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":97648},{"type":"v1","addr":"172.21.15.191:6811","nonce":97648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":97648},{"type":"v1","addr":"172.21.15.191:6815","nonce":97648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":97648},{"type":"v1","addr":"172.21.15.191:6813","nonce":97648}]},"public_addr":"172.21.15.191:6809/97648","cluster_addr":"172.21.15.191:6811/97648","heartbeat_back_addr":"172.21.15.191:6815/97648","heartbeat_front_addr":"172.21.15.191:6813/97648","state":["exists","up"]},{"osd":3,"uuid":"b4483b81-cc61-477a-ada6-4620185b9f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":97649},{"type":"v1","addr":"172.21.15.191:6801","nonce":97649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":97649},{"type":"v1","addr":"172.21.15.191:6803","nonce":97649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":97649},{"type":"v1","addr":"172.21.15.191:6807","nonce":97649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":97649},{"type":"v1","addr":"172.21.15.191:6805","nonce":97649}]},"public_addr":"172.21.15.191:6801/97649","cluster_addr":"172.21.15.191:6803/97649","heartbeat_back_addr":"172.21.15.191:6807/97649","heartbeat_front_addr":"172.21.15.191:6805/97649","state":["exists","up"]},{"osd":4,"uuid":"f133b7f6-ca18-4ff3-a71f-d9ae1593ce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":97900},{"type":"v1","addr":"172.21.15.196:6809","nonce":97900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":97900},{"type":"v1","addr":"172.21.15.196:6811","nonce":97900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":97900},{"type":"v1","addr":"172.21.15.196:6815","nonce":97900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":97900},{"type":"v1","addr":"172.21.15.196:6813","nonce":97900}]},"public_addr":"172.21.15.196:6809/97900","cluster_addr":"172.21.15.196:6811/97900","heartbeat_back_addr":"172.21.15.196:6815/97900","heartbeat_front_addr":"172.21.15.196:6813/97900","state":["exists","up"]},{"osd":5,"uuid":"b284d515-2796-4d33-81a8-0e42195d49bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":97901},{"type":"v1","addr":"172.21.15.196:6801","nonce":97901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":97901},{"type":"v1","addr":"172.21.15.196:6803","nonce":97901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":97901},{"type":"v1","addr":"172.21.15.196:6807","nonce":97901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":97901},{"type":"v1","addr":"172.21.15.196:6805","nonce":97901}]},"public_addr":"172.21.15.196:6801/97901","cluster_addr":"172.21.15.196:6803/97901","heartbeat_back_addr":"172.21.15.196:6807/97901","heartbeat_front_addr":"172.21.15.196:6805/97901","state":["exists","up"]},{"osd":6,"uuid":"37ee78d6-7798-4d17-ad11-4fffd8355d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":97902},{"type":"v1","addr":"172.21.15.196:6817","nonce":97902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":97902},{"type":"v1","addr":"172.21.15.196:6819","nonce":97902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":97902},{"type":"v1","addr":"172.21.15.196:6823","nonce":97902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":97902},{"type":"v1","addr":"172.21.15.196:6821","nonce":97902}]},"public_addr":"172.21.15.196:6817/97902","cluster_addr":"172.21.15.196:6819/97902","heartbeat_back_addr":"172.21.15.196:6823/97902","heartbeat_front_addr":"172.21.15.196:6821/97902","state":["exists","up"]},{"osd":7,"uuid":"0c1c2708-0d59-4f5b-89aa-75ea67df828e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":97907},{"type":"v1","addr":"172.21.15.196:6825","nonce":97907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":97907},{"type":"v1","addr":"172.21.15.196:6827","nonce":97907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":97907},{"type":"v1","addr":"172.21.15.196:6831","nonce":97907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":97907},{"type":"v1","addr":"172.21.15.196:6829","nonce":97907}]},"public_addr":"172.21.15.196:6825/97907","cluster_addr":"172.21.15.196:6827/97907","heartbeat_back_addr":"172.21.15.196:6831/97907","heartbeat_front_addr":"172.21.15.196:6829/97907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.195233+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.178597+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.205385+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.186833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.213154+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.109613+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.150885+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.215448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-24T23:19:52.548 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.547+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f7da4069de0 msgr2=0x7f7da406c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.548 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.547+0000 7f7dc816f700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f7da4069de0 0x7f7da406c290 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7dc0160400 tx=0x7f7dbc009040 comp rx=0 tx=0).stop 2023-03-24T23:19:52.548 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.547+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc006f030 msgr2=0x7f7dc015fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.548 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.547+0000 7f7dc816f700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc006f030 0x7f7dc015fc40 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7db000c700 tx=0x7f7db000cac0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.548 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.548+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 shutdown_connections 2023-03-24T23:19:52.549 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.548+0000 7f7dc816f700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f7da4069de0 0x7f7da406c290 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.549 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.548+0000 7f7dc816f700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f7dc006f030 0x7f7dc015fc40 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.549 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.548+0000 7f7dc816f700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7dc015c990 0x7f7dc00c70e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.549 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.548+0000 7f7dc816f700 1 --2- 172.21.15.191:0/374793677 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7dc006f4d0 0x7f7dc0146e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.549 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.548+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 >> 172.21.15.191:0/374793677 conn(0x7f7dc00cc8f0 msgr2=0x7f7dc00b0790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:52.549 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.548+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 shutdown_connections 2023-03-24T23:19:52.549 INFO:teuthology.misc.health.smithi191.stderr:2023-03-24T23:19:52.548+0000 7f7dc816f700 1 -- 172.21.15.191:0/374793677 wait complete. 2023-03-24T23:19:52.559 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-03-24T23:19:52.559 INFO:tasks.ceph:Creating RBD pool 2023-03-24T23:19:52.559 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd pool create rbd 8 2023-03-24T23:19:52.678 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.678+0000 7f141556d700 1 Processor -- start 2023-03-24T23:19:52.678 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.678+0000 7f141556d700 1 -- start start 2023-03-24T23:19:52.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.678+0000 7f141556d700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f141006c010 0x7f141006ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.678+0000 7f141556d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14100cba00 con 0x7f141015cab0 2023-03-24T23:19:52.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.678+0000 7f141556d700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14100cbce0 con 0x7f141006ca90 2023-03-24T23:19:52.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.678+0000 7f141556d700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f141006c460 con 0x7f141006c010 2023-03-24T23:19:52.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140effd700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f141015cab0 0x7f14100cb120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57782/0 (socket says 172.21.15.191:57782) 2023-03-24T23:19:52.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140effd700 1 -- 172.21.15.191:0/3267508827 learned_addr learned my addr 172.21.15.191:0/3267508827 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:52.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140e7fc700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f141006c010 0x7f141006ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294469479 0 0) 0x7f14100cba00 con 0x7f141015cab0 2023-03-24T23:19:52.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13f80035b0 con 0x7f141015cab0 2023-03-24T23:19:52.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292579165 0 0) 0x7f14100cbce0 con 0x7f141006ca90 2023-03-24T23:19:52.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14100cba00 con 0x7f141006ca90 2023-03-24T23:19:52.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 <== mon.0 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2648944549 0 0) 0x7f13f80035b0 con 0x7f141015cab0 2023-03-24T23:19:52.682 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f141006c010 msgr2=0x7f141006ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:52.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f141006c010 0x7f141006ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 >> v1:172.21.15.196:6789/0 conn(0x7f141006ca90 legacy=0x7f141006cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141006a5a0 con 0x7f141015cab0 2023-03-24T23:19:52.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 <== mon.0 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1537907258 0 0) 0x7f1400001670 con 0x7f141015cab0 2023-03-24T23:19:52.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 >> v1:172.21.15.191:6789/0 conn(0x7f141015cab0 legacy=0x7f14100cb120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f8003bc0 0x7f13f8003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f141006c010 0x7f13f8004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13f8004d90 0x7f13f8005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141006c460 con 0x7f13f8003bc0 2023-03-24T23:19:52.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f800bac0 con 0x7f13f8004d90 2023-03-24T23:19:52.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f800bc30 con 0x7f141006c010 2023-03-24T23:19:52.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140effd700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f8003bc0 0x7f13f8003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140e7fc700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f141006c010 0x7f13f8004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140f7fe700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13f8004d90 0x7f13f8005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140effd700 1 -- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f141006c010 msgr2=0x7f13f8004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:52.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140effd700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f141006c010 0x7f13f8004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140effd700 1 -- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13f8004d90 msgr2=0x7f13f8005200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:52.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140effd700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f13f8004d90 0x7f13f8005200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.679+0000 7f140effd700 1 -- 172.21.15.191:0/3267508827 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f80035b0 con 0x7f13f8003bc0 2023-03-24T23:19:52.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f140effd700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f8003bc0 0x7f13f8003fb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f140000c240 tx=0x7f13f8005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8e23d65f6d46071 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:52.685 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140000cd20 con 0x7f13f8003bc0 2023-03-24T23:19:52.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1400005960 con 0x7f13f8003bc0 2023-03-24T23:19:52.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f140dffb700 1 -- 172.21.15.191:0/3267508827 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140000cea0 con 0x7f13f8003bc0 2023-03-24T23:19:52.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f141556d700 1 -- 172.21.15.191:0/3267508827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f8003bc0 msgr2=0x7f13f8003fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f141556d700 1 --2- 172.21.15.191:0/3267508827 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13f8003bc0 0x7f13f8003fb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f140000c240 tx=0x7f13f8005880 comp rx=0 tx=0).stop 2023-03-24T23:19:52.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f140f7fe700 1 -- 172.21.15.191:0/3267508827 reap_dead start 2023-03-24T23:19:52.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f141556d700 1 -- 172.21.15.191:0/3267508827 shutdown_connections 2023-03-24T23:19:52.686 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f141556d700 1 -- 172.21.15.191:0/3267508827 >> 172.21.15.191:0/3267508827 conn(0x7f14100d0390 msgr2=0x7f14100cc180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:52.687 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f141556d700 1 -- 172.21.15.191:0/3267508827 shutdown_connections 2023-03-24T23:19:52.687 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.680+0000 7f141556d700 1 -- 172.21.15.191:0/3267508827 wait complete. 2023-03-24T23:19:52.689 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.681+0000 7f141556d700 1 Processor -- start 2023-03-24T23:19:52.689 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f141556d700 1 -- start start 2023-03-24T23:19:52.689 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f141556d700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13f8003bc0 0x7f14101512d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.689 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f141556d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14101520e0 0x7f14100b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f141556d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14100c7020 0x7f14100cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f141556d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14100b44b0 con 0x7f14101520e0 2023-03-24T23:19:52.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f141556d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14100b3ef0 con 0x7f14100c7020 2023-03-24T23:19:52.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f141556d700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14100b41d0 con 0x7f13f8003bc0 2023-03-24T23:19:52.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f140e7fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14101520e0 0x7f14100b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f140e7fc700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14101520e0 0x7f14100b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50396/0 (socket says 172.21.15.191:50396) 2023-03-24T23:19:52.690 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f140e7fc700 1 -- 172.21.15.191:0/3578502869 learned_addr learned my addr 172.21.15.191:0/3578502869 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:52.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f140f7fe700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14100c7020 0x7f14100cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.682+0000 7f140effd700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13f8003bc0 0x7f14101512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f140e7fc700 1 -- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13f8003bc0 msgr2=0x7f14101512d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f140e7fc700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13f8003bc0 0x7f14101512d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f140e7fc700 1 -- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14100c7020 msgr2=0x7f14100cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f140e7fc700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14100c7020 0x7f14100cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f140e7fc700 1 -- 172.21.15.191:0/3578502869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f141015ce80 con 0x7f14101520e0 2023-03-24T23:19:52.691 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f140effd700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13f8003bc0 0x7f14101512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-24T23:19:52.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f140e7fc700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14101520e0 0x7f14100b3580 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1404011aa0 tx=0x7f1404011db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:52.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f13f77fe700 1 -- 172.21.15.191:0/3578502869 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140400f940 con 0x7f14101520e0 2023-03-24T23:19:52.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f13f77fe700 1 -- 172.21.15.191:0/3578502869 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f140400faa0 con 0x7f14101520e0 2023-03-24T23:19:52.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141006a880 con 0x7f14101520e0 2023-03-24T23:19:52.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f13f77fe700 1 -- 172.21.15.191:0/3578502869 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f140400fe50 con 0x7f14101520e0 2023-03-24T23:19:52.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.683+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141015d010 con 0x7f14101520e0 2023-03-24T23:19:52.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.684+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14100b0a70 con 0x7f14101520e0 2023-03-24T23:19:52.692 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.684+0000 7f13f77fe700 1 -- 172.21.15.191:0/3578502869 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f1404014440 con 0x7f14101520e0 2023-03-24T23:19:52.693 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.684+0000 7f13f77fe700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f13fc069de0 0x7f13fc06c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:52.693 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.685+0000 7f13f77fe700 1 -- 172.21.15.191:0/3578502869 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f1404088250 con 0x7f14101520e0 2023-03-24T23:19:52.693 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.685+0000 7f140effd700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f13fc069de0 0x7f13fc06c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:52.693 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.685+0000 7f140effd700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f13fc069de0 0x7f13fc06c290 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1400000c00 tx=0x7f1400007f90 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:52.693 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.689+0000 7f13f77fe700 1 -- 172.21.15.191:0/3578502869 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f1404057aa0 con 0x7f14101520e0 2023-03-24T23:19:52.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.875+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f141006ec60 con 0x7f14101520e0 2023-03-24T23:19:52.988 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.988+0000 7f13f77fe700 1 -- 172.21.15.191:0/3578502869 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) v1 ==== 109+0+0 (secure 0 0 0) 0x7f1404057630 con 0x7f14101520e0 2023-03-24T23:19:52.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f13fc069de0 msgr2=0x7f13fc06c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f13fc069de0 0x7f13fc06c290 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1400000c00 tx=0x7f1400007f90 comp rx=0 tx=0).stop 2023-03-24T23:19:52.992 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14101520e0 msgr2=0x7f14100b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:52.992 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14101520e0 0x7f14100b3580 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f1404011aa0 tx=0x7f1404011db0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.992 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 shutdown_connections 2023-03-24T23:19:52.992 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f13fc069de0 0x7f13fc06c290 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.992 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14100c7020 0x7f14100cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.992 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13f8003bc0 0x7f14101512d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.992 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 --2- 172.21.15.191:0/3578502869 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f14101520e0 0x7f14100b3580 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 >> 172.21.15.191:0/3578502869 conn(0x7f14100d0390 msgr2=0x7f1410071510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 shutdown_connections 2023-03-24T23:19:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:52.991+0000 7f141556d700 1 -- 172.21.15.191:0/3578502869 wait complete. 2023-03-24T23:19:52.993 INFO:teuthology.orchestra.run.smithi191.stderr:pool 'rbd' created 2023-03-24T23:19:53.003 DEBUG:teuthology.orchestra.run.smithi191:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-03-24T23:19:53.124 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.124+0000 7f0a22a8d700 1 Processor -- start 2023-03-24T23:19:53.125 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.124+0000 7f0a22a8d700 1 -- start start 2023-03-24T23:19:53.127 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a22a8d700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c15d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:53.127 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a22a8d700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1c160610 con 0x7f0a1c0ced00 2023-03-24T23:19:53.127 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a22a8d700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a1c1608f0 con 0x7f0a1c0cf7e0 2023-03-24T23:19:53.127 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a22a8d700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c0cf170 con 0x7f0a1c0716f0 2023-03-24T23:19:53.127 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a1bfff700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f0a1c0ced00 0x7f0a1c070fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57802/0 (socket says 172.21.15.191:57802) 2023-03-24T23:19:53.128 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a1bfff700 1 -- 172.21.15.191:0/1810142793 learned_addr learned my addr 172.21.15.191:0/1810142793 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:53.128 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a2102a700 1 --2- 172.21.15.191:0/1810142793 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c15d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:53.128 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a1b7fe700 1 -- 172.21.15.191:0/1810142793 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302172413 0 0) 0x7f0a1c160610 con 0x7f0a1c0ced00 2023-03-24T23:19:53.128 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a1b7fe700 1 -- 172.21.15.191:0/1810142793 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c0035b0 con 0x7f0a1c0ced00 2023-03-24T23:19:53.128 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a1b7fe700 1 -- 172.21.15.191:0/1810142793 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771485185 0 0) 0x7f0a1c1608f0 con 0x7f0a1c0cf7e0 2023-03-24T23:19:53.128 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a2102a700 1 -- 172.21.15.191:0/1810142793 >> v1:172.21.15.196:6789/0 conn(0x7f0a1c0cf7e0 legacy=0x7f0a1c0cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:53.128 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a2102a700 1 -- 172.21.15.191:0/1810142793 >> v1:172.21.15.191:6789/0 conn(0x7f0a1c0ced00 legacy=0x7f0a1c070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:53.129 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.125+0000 7f0a2102a700 1 -- 172.21.15.191:0/1810142793 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a1c160bd0 con 0x7f0a1c0716f0 2023-03-24T23:19:53.129 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a2102a700 1 --2- 172.21.15.191:0/1810142793 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c15d370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009dc0 tx=0x7f0a1400b330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fcf98adcd237e2b server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:53.129 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a1b7fe700 1 -- 172.21.15.191:0/1810142793 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a1400e070 con 0x7f0a1c0716f0 2023-03-24T23:19:53.129 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a1b7fe700 1 -- 172.21.15.191:0/1810142793 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a14003a20 con 0x7f0a1c0716f0 2023-03-24T23:19:53.129 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a1b7fe700 1 -- 172.21.15.191:0/1810142793 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08001670 con 0x7f0a1c0716f0 2023-03-24T23:19:53.130 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a22a8d700 1 -- 172.21.15.191:0/1810142793 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 msgr2=0x7f0a1c15d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:53.130 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a22a8d700 1 --2- 172.21.15.191:0/1810142793 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c15d370 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009dc0 tx=0x7f0a1400b330 comp rx=0 tx=0).stop 2023-03-24T23:19:53.130 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a22a8d700 1 -- 172.21.15.191:0/1810142793 shutdown_connections 2023-03-24T23:19:53.130 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a22a8d700 1 --2- 172.21.15.191:0/1810142793 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c15d370 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:53.130 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a22a8d700 1 -- 172.21.15.191:0/1810142793 >> 172.21.15.191:0/1810142793 conn(0x7f0a1c0d0b50 msgr2=0x7f0a1c0cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:53.130 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a22a8d700 1 -- 172.21.15.191:0/1810142793 shutdown_connections 2023-03-24T23:19:53.130 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.126+0000 7f0a22a8d700 1 -- 172.21.15.191:0/1810142793 wait complete. 2023-03-24T23:19:53.133 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.127+0000 7f0a22a8d700 1 Processor -- start 2023-03-24T23:19:53.133 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a22a8d700 1 -- start start 2023-03-24T23:19:53.133 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a22a8d700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c0c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:53.133 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a22a8d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0ced00 0x7f0a1c1515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:53.133 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a22a8d700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a1c0cf7e0 0x7f0a1c1f4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:53.133 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a22a8d700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c1529a0 con 0x7f0a1c0ced00 2023-03-24T23:19:53.134 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a22a8d700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c151af0 con 0x7f0a1c0cf7e0 2023-03-24T23:19:53.134 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a22a8d700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a1c1f5470 con 0x7f0a1c0716f0 2023-03-24T23:19:53.134 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a1bfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0ced00 0x7f0a1c1515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:53.134 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.128+0000 7f0a1bfff700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0ced00 0x7f0a1c1515b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50408/0 (socket says 172.21.15.191:50408) 2023-03-24T23:19:53.134 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a1bfff700 1 -- 172.21.15.191:0/3892468076 learned_addr learned my addr 172.21.15.191:0/3892468076 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:53.134 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a20829700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c0c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:53.134 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a2102a700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a1c0cf7e0 0x7f0a1c1f4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:53.134 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a1bfff700 1 -- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 msgr2=0x7f0a1c0c5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:53.135 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a1bfff700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c0c5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:53.135 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a1bfff700 1 -- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a1c0cf7e0 msgr2=0x7f0a1c1f4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:53.135 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a1bfff700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a1c0cf7e0 0x7f0a1c1f4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:53.135 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a1bfff700 1 -- 172.21.15.191:0/3892468076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a14009a70 con 0x7f0a1c0ced00 2023-03-24T23:19:53.135 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a2102a700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a1c0cf7e0 0x7f0a1c1f4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:53.135 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a1bfff700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0ced00 0x7f0a1c1515b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800c750 tx=0x7f0a0800cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:53.135 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a197fa700 1 -- 172.21.15.191:0/3892468076 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08004c80 con 0x7f0a1c0ced00 2023-03-24T23:19:53.136 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a197fa700 1 -- 172.21.15.191:0/3892468076 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a08004de0 con 0x7f0a1c0ced00 2023-03-24T23:19:53.136 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a1c161190 con 0x7f0a1c0ced00 2023-03-24T23:19:53.136 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a197fa700 1 -- 172.21.15.191:0/3892468076 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a08007310 con 0x7f0a1c0ced00 2023-03-24T23:19:53.136 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.129+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a1c160eb0 con 0x7f0a1c0ced00 2023-03-24T23:19:53.136 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.130+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a1c161470 con 0x7f0a1c0ced00 2023-03-24T23:19:53.136 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.131+0000 7f0a197fa700 1 -- 172.21.15.191:0/3892468076 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f0a08007470 con 0x7f0a1c0ced00 2023-03-24T23:19:53.136 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.131+0000 7f0a197fa700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f0a00069de0 0x7f0a0006c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:53.137 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.131+0000 7f0a197fa700 1 -- 172.21.15.191:0/3892468076 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5199+0+0 (secure 0 0 0) 0x7f0a080872a0 con 0x7f0a1c0ced00 2023-03-24T23:19:53.137 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.131+0000 7f0a20829700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f0a00069de0 0x7f0a0006c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:53.137 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.131+0000 7f0a20829700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f0a00069de0 0x7f0a0006c290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0a10000f60 tx=0x7f0a10009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:53.137 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.135+0000 7f0a197fa700 1 -- 172.21.15.191:0/3892468076 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f0a08052eb0 con 0x7f0a1c0ced00 2023-03-24T23:19:53.325 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.325+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0a1c0ceae0 con 0x7f0a1c0ced00 2023-03-24T23:19:53.986 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.986+0000 7f0a197fa700 1 -- 172.21.15.191:0/3892468076 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v18) v1 ==== 173+0+0 (secure 0 0 0) 0x7f0a080564d0 con 0x7f0a1c0ced00 2023-03-24T23:19:53.990 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f0a00069de0 msgr2=0x7f0a0006c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:53.990 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f0a00069de0 0x7f0a0006c290 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0a10000f60 tx=0x7f0a10009040 comp rx=0 tx=0).stop 2023-03-24T23:19:53.990 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0ced00 msgr2=0x7f0a1c1515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:53.990 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0ced00 0x7f0a1c1515b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800c750 tx=0x7f0a0800cb10 comp rx=0 tx=0).stop 2023-03-24T23:19:53.990 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 shutdown_connections 2023-03-24T23:19:53.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f0a00069de0 0x7f0a0006c290 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:53.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a1c0cf7e0 0x7f0a1c1f4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:53.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a1c0716f0 0x7f0a1c0c5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:53.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 --2- 172.21.15.191:0/3892468076 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f0a1c0ced00 0x7f0a1c1515b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:53.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.989+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 >> 172.21.15.191:0/3892468076 conn(0x7f0a1c0d0b50 msgr2=0x7f0a1c0b04a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:53.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.990+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 shutdown_connections 2023-03-24T23:19:53.991 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:53.990+0000 7f0a22a8d700 1 -- 172.21.15.191:0/3892468076 wait complete. 2023-03-24T23:19:53.992 INFO:teuthology.orchestra.run.smithi191.stderr:enabled application 'rbd' on pool 'rbd' 2023-03-24T23:19:54.003 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-03-24T23:19:54.003 INFO:tasks.ceph.mds.a:Restarting daemon 2023-03-24T23:19:54.004 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-03-24T23:19:54.006 INFO:tasks.ceph.mds.a:Started 2023-03-24T23:19:54.006 INFO:tasks.ceph.mds.c:Restarting daemon 2023-03-24T23:19:54.007 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-03-24T23:19:54.009 INFO:tasks.ceph.mds.c:Started 2023-03-24T23:19:54.009 INFO:tasks.ceph.mds.b:Restarting daemon 2023-03-24T23:19:54.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-03-24T23:19:54.012 INFO:tasks.ceph.mds.b:Started 2023-03-24T23:19:54.012 INFO:tasks.ceph.mds.d:Restarting daemon 2023-03-24T23:19:54.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-03-24T23:19:54.014 INFO:tasks.ceph.mds.d:Started 2023-03-24T23:19:54.015 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-03-24T23:19:54.015 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-24T23:19:54.085 INFO:tasks.ceph.mds.a.smithi191.stdout:starting mds.a at 2023-03-24T23:19:54.086 INFO:tasks.ceph.mds.c.smithi191.stdout:starting mds.c at 2023-03-24T23:19:54.099 INFO:tasks.ceph.mds.b.smithi196.stdout:starting mds.b at 2023-03-24T23:19:54.099 INFO:tasks.ceph.mds.d.smithi196.stdout:starting mds.d at 2023-03-24T23:19:54.144 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.144+0000 7f68eb50e700 1 Processor -- start 2023-03-24T23:19:54.144 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.144+0000 7f68eb50e700 1 -- start start 2023-03-24T23:19:54.146 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68eb50e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f68e415a960 0x7f68e415b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.146 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68eb50e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e4071050 0x7f68e40ca540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.147 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68eb50e700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68e415be90 con 0x7f68e40caa80 2023-03-24T23:19:54.147 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68eb50e700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68e40b3100 con 0x7f68e415a960 2023-03-24T23:19:54.147 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68eb50e700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e415ad30 con 0x7f68e4071050 2023-03-24T23:19:54.147 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e92aa700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f68e415a960 0x7f68e415b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.147 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e92aa700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f68e415a960 0x7f68e415b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50440/0 (socket says 172.21.15.191:50440) 2023-03-24T23:19:54.147 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e9aab700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f68e40caa80 0x7f68e415e4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57844/0 (socket says 172.21.15.191:57844) 2023-03-24T23:19:54.147 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e92aa700 1 -- 172.21.15.191:0/302525902 learned_addr learned my addr 172.21.15.191:0/302525902 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e8aa9700 1 --2- 172.21.15.191:0/302525902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e4071050 0x7f68e40ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e92aa700 1 -- 172.21.15.191:0/302525902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e4071050 msgr2=0x7f68e40ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e92aa700 1 --2- 172.21.15.191:0/302525902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e4071050 0x7f68e40ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e92aa700 1 -- 172.21.15.191:0/302525902 >> v1:172.21.15.191:6789/0 conn(0x7f68e40caa80 legacy=0x7f68e415e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68dbfff700 1 -- 172.21.15.191:0/302525902 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137677359 0 0) 0x7f68e415be90 con 0x7f68e40caa80 2023-03-24T23:19:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e92aa700 1 -- 172.21.15.191:0/302525902 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e40b2d40 con 0x7f68e415a960 2023-03-24T23:19:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68e92aa700 1 --2- 172.21.15.191:0/302525902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e415a960 0x7f68e415b950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f68e00098f0 tx=0x7f68e0009c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ddf74347b0ed4f8 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.148 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68dbfff700 1 -- 172.21.15.191:0/302525902 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0011830 con 0x7f68e415a960 2023-03-24T23:19:54.149 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68dbfff700 1 -- 172.21.15.191:0/302525902 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e0011990 con 0x7f68e415a960 2023-03-24T23:19:54.149 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.145+0000 7f68dbfff700 1 -- 172.21.15.191:0/302525902 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0011d20 con 0x7f68e415a960 2023-03-24T23:19:54.149 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.146+0000 7f68eb50e700 1 -- 172.21.15.191:0/302525902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e415a960 msgr2=0x7f68e415b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.149 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.146+0000 7f68eb50e700 1 --2- 172.21.15.191:0/302525902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e415a960 0x7f68e415b950 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f68e00098f0 tx=0x7f68e0009c00 comp rx=0 tx=0).stop 2023-03-24T23:19:54.149 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.146+0000 7f68eb50e700 1 -- 172.21.15.191:0/302525902 shutdown_connections 2023-03-24T23:19:54.149 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.146+0000 7f68eb50e700 1 --2- 172.21.15.191:0/302525902 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e415a960 0x7f68e415b950 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.149 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.146+0000 7f68eb50e700 1 --2- 172.21.15.191:0/302525902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e4071050 0x7f68e40ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.150 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.146+0000 7f68eb50e700 1 -- 172.21.15.191:0/302525902 >> 172.21.15.191:0/302525902 conn(0x7f68e40cfef0 msgr2=0x7f68e40d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:54.150 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.146+0000 7f68eb50e700 1 -- 172.21.15.191:0/302525902 shutdown_connections 2023-03-24T23:19:54.150 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.146+0000 7f68eb50e700 1 -- 172.21.15.191:0/302525902 wait complete. 2023-03-24T23:19:54.152 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.147+0000 7f68eb50e700 1 Processor -- start 2023-03-24T23:19:54.152 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.147+0000 7f68eb50e700 1 -- start start 2023-03-24T23:19:54.152 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68eb50e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e4071050 0x7f68e40c5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.152 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68eb50e700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e40caa80 0x7f68e415d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.152 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68eb50e700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68e415a960 0x7f68e414e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.152 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68eb50e700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e40c62c0 con 0x7f68e4071050 2023-03-24T23:19:54.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68eb50e700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e415db10 con 0x7f68e415a960 2023-03-24T23:19:54.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68eb50e700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68e414eb50 con 0x7f68e40caa80 2023-03-24T23:19:54.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e4071050 0x7f68e40c5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e4071050 0x7f68e40c5390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50448/0 (socket says 172.21.15.191:50448) 2023-03-24T23:19:54.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 -- 172.21.15.191:0/304604246 learned_addr learned my addr 172.21.15.191:0/304604246 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:54.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e8aa9700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e40caa80 0x7f68e415d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e9aab700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68e415a960 0x7f68e414e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 -- 172.21.15.191:0/304604246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e40caa80 msgr2=0x7f68e415d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e40caa80 0x7f68e415d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 -- 172.21.15.191:0/304604246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68e415a960 msgr2=0x7f68e414e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68e415a960 0x7f68e414e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 -- 172.21.15.191:0/304604246 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68e0009550 con 0x7f68e4071050 2023-03-24T23:19:54.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.148+0000 7f68e92aa700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e4071050 0x7f68e40c5390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009c40 tx=0x7f68e0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.149+0000 7f68d9ffb700 1 -- 172.21.15.191:0/304604246 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e0004300 con 0x7f68e4071050 2023-03-24T23:19:54.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.149+0000 7f68d9ffb700 1 -- 172.21.15.191:0/304604246 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e001f070 con 0x7f68e4071050 2023-03-24T23:19:54.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.149+0000 7f68d9ffb700 1 -- 172.21.15.191:0/304604246 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68e00036a0 con 0x7f68e4071050 2023-03-24T23:19:54.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.149+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e40718b0 con 0x7f68e4071050 2023-03-24T23:19:54.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.149+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e40715d0 con 0x7f68e4071050 2023-03-24T23:19:54.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.150+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e4071b90 con 0x7f68e4071050 2023-03-24T23:19:54.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.150+0000 7f68d9ffb700 1 -- 172.21.15.191:0/304604246 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f68e0021030 con 0x7f68e4071050 2023-03-24T23:19:54.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.151+0000 7f68d9ffb700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f68c8069de0 0x7f68c806c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.151+0000 7f68e8aa9700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f68c8069de0 0x7f68c806c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.151+0000 7f68d9ffb700 1 -- 172.21.15.191:0/304604246 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f68e0088fe0 con 0x7f68e4071050 2023-03-24T23:19:54.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.151+0000 7f68e8aa9700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f68c8069de0 0x7f68c806c290 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f68e40adb60 tx=0x7f68d4009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.155+0000 7f68d9ffb700 1 -- 172.21.15.191:0/304604246 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f68e0054bf0 con 0x7f68e4071050 2023-03-24T23:19:54.346 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.346+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68e4074580 con 0x7f68e4071050 2023-03-24T23:19:54.347 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.347+0000 7f68d9ffb700 1 -- 172.21.15.191:0/304604246 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v18) v1 ==== 74+0+14255 (secure 0 0 0) 0x7f68e0058210 con 0x7f68e4071050 2023-03-24T23:19:54.347 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:54.347 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":18,"fsid":"72b355de-1f4b-4f02-899f-839bc265bc0f","created":"2023-03-24T23:19:34.494902+0000","modified":"2023-03-24T23:19:53.982844+0000","last_up_change":"2023-03-24T23:19:45.928145+0000","last_in_change":"2023-03-24T23:19:36.946912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-24T23:19:46.103710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-24T23:19:52.893681+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ca3720a-fe6e-4b70-a32f-fef43c5da336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6817","nonce":97650},{"type":"v1","addr":"172.21.15.191:6819","nonce":97650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6821","nonce":97650},{"type":"v1","addr":"172.21.15.191:6823","nonce":97650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6829","nonce":97650},{"type":"v1","addr":"172.21.15.191:6831","nonce":97650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":97650},{"type":"v1","addr":"172.21.15.191:6827","nonce":97650}]},"public_addr":"172.21.15.191:6819/97650","cluster_addr":"172.21.15.191:6823/97650","heartbeat_back_addr":"172.21.15.191:6831/97650","heartbeat_front_addr":"172.21.15.191:6827/97650","state":["exists","up"]},{"osd":1,"uuid":"55f40987-e709-41aa-84f1-7a0b426e984d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":97651},{"type":"v1","addr":"172.21.15.191:6818","nonce":97651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":97651},{"type":"v1","addr":"172.21.15.191:6822","nonce":97651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":97651},{"type":"v1","addr":"172.21.15.191:6830","nonce":97651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":97651},{"type":"v1","addr":"172.21.15.191:6826","nonce":97651}]},"public_addr":"172.21.15.191:6818/97651","cluster_addr":"172.21.15.191:6822/97651","heartbeat_back_addr":"172.21.15.191:6830/97651","heartbeat_front_addr":"172.21.15.191:6826/97651","state":["exists","up"]},{"osd":2,"uuid":"b2e378f2-4cdb-4900-8f17-83affc66d8ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":97648},{"type":"v1","addr":"172.21.15.191:6809","nonce":97648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":97648},{"type":"v1","addr":"172.21.15.191:6811","nonce":97648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":97648},{"type":"v1","addr":"172.21.15.191:6815","nonce":97648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":97648},{"type":"v1","addr":"172.21.15.191:6813","nonce":97648}]},"public_addr":"172.21.15.191:6809/97648","cluster_addr":"172.21.15.191:6811/97648","heartbeat_back_addr":"172.21.15.191:6815/97648","heartbeat_front_addr":"172.21.15.191:6813/97648","state":["exists","up"]},{"osd":3,"uuid":"b4483b81-cc61-477a-ada6-4620185b9f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":97649},{"type":"v1","addr":"172.21.15.191:6801","nonce":97649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":97649},{"type":"v1","addr":"172.21.15.191:6803","nonce":97649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":97649},{"type":"v1","addr":"172.21.15.191:6807","nonce":97649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":97649},{"type":"v1","addr":"172.21.15.191:6805","nonce":97649}]},"public_addr":"172.21.15.191:6801/97649","cluster_addr":"172.21.15.191:6803/97649","heartbeat_back_addr":"172.21.15.191:6807/97649","heartbeat_front_addr":"172.21.15.191:6805/97649","state":["exists","up"]},{"osd":4,"uuid":"f133b7f6-ca18-4ff3-a71f-d9ae1593ce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":97900},{"type":"v1","addr":"172.21.15.196:6809","nonce":97900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":97900},{"type":"v1","addr":"172.21.15.196:6811","nonce":97900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":97900},{"type":"v1","addr":"172.21.15.196:6815","nonce":97900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":97900},{"type":"v1","addr":"172.21.15.196:6813","nonce":97900}]},"public_addr":"172.21.15.196:6809/97900","cluster_addr":"172.21.15.196:6811/97900","heartbeat_back_addr":"172.21.15.196:6815/97900","heartbeat_front_addr":"172.21.15.196:6813/97900","state":["exists","up"]},{"osd":5,"uuid":"b284d515-2796-4d33-81a8-0e42195d49bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":97901},{"type":"v1","addr":"172.21.15.196:6801","nonce":97901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":97901},{"type":"v1","addr":"172.21.15.196:6803","nonce":97901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":97901},{"type":"v1","addr":"172.21.15.196:6807","nonce":97901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":97901},{"type":"v1","addr":"172.21.15.196:6805","nonce":97901}]},"public_addr":"172.21.15.196:6801/97901","cluster_addr":"172.21.15.196:6803/97901","heartbeat_back_addr":"172.21.15.196:6807/97901","heartbeat_front_addr":"172.21.15.196:6805/97901","state":["exists","up"]},{"osd":6,"uuid":"37ee78d6-7798-4d17-ad11-4fffd8355d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":97902},{"type":"v1","addr":"172.21.15.196:6817","nonce":97902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":97902},{"type":"v1","addr":"172.21.15.196:6819","nonce":97902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":97902},{"type":"v1","addr":"172.21.15.196:6823","nonce":97902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":97902},{"type":"v1","addr":"172.21.15.196:6821","nonce":97902}]},"public_addr":"172.21.15.196:6817/97902","cluster_addr":"172.21.15.196:6819/97902","heartbeat_back_addr":"172.21.15.196:6823/97902","heartbeat_front_addr":"172.21.15.196:6821/97902","state":["exists","up"]},{"osd":7,"uuid":"0c1c2708-0d59-4f5b-89aa-75ea67df828e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":97907},{"type":"v1","addr":"172.21.15.196:6825","nonce":97907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":97907},{"type":"v1","addr":"172.21.15.196:6827","nonce":97907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":97907},{"type":"v1","addr":"172.21.15.196:6831","nonce":97907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":97907},{"type":"v1","addr":"172.21.15.196:6829","nonce":97907}]},"public_addr":"172.21.15.196:6825/97907","cluster_addr":"172.21.15.196:6827/97907","heartbeat_back_addr":"172.21.15.196:6831/97907","heartbeat_front_addr":"172.21.15.196:6829/97907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.195233+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.178597+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.205385+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.186833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.213154+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.109613+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.150885+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.215448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-24T23:19:54.349 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f68c8069de0 msgr2=0x7f68c806c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.349 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f68c8069de0 0x7f68c806c290 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f68e40adb60 tx=0x7f68d4009210 comp rx=0 tx=0).stop 2023-03-24T23:19:54.350 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e4071050 msgr2=0x7f68e40c5390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.350 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e4071050 0x7f68e40c5390 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f68e0009c40 tx=0x7f68e0004750 comp rx=0 tx=0).stop 2023-03-24T23:19:54.350 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 shutdown_connections 2023-03-24T23:19:54.350 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f68c8069de0 0x7f68c806c290 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.350 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f68e4071050 0x7f68e40c5390 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.350 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f68e415a960 0x7f68e414e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.350 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 --2- 172.21.15.191:0/304604246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f68e40caa80 0x7f68e415d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.351 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 >> 172.21.15.191:0/304604246 conn(0x7f68e40cfef0 msgr2=0x7f68e40b2050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:54.351 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 shutdown_connections 2023-03-24T23:19:54.351 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.349+0000 7f68eb50e700 1 -- 172.21.15.191:0/304604246 wait complete. 2023-03-24T23:19:54.362 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-24T23:19:46.103710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-24T23:19:52.893681+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-24T23:19:54.362 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-24T23:19:54.484 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.484+0000 7fee37aec700 1 Processor -- start 2023-03-24T23:19:54.485 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.484+0000 7fee37aec700 1 -- start start 2023-03-24T23:19:54.487 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee37aec700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee30070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.487 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee37aec700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fee30071520 0x7fee3015d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.487 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee37aec700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee30160610 con 0x7fee300cf7e0 2023-03-24T23:19:54.487 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee37aec700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee300cf170 con 0x7fee30071520 2023-03-24T23:19:54.487 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee37aec700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee300719a0 con 0x7fee300ced00 2023-03-24T23:19:54.487 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee36089700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fee30071520 0x7fee3015d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.488 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee36089700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fee30071520 0x7fee3015d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50458/0 (socket says 172.21.15.191:50458) 2023-03-24T23:19:54.488 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee35888700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fee300cf7e0 0x7fee300cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57854/0 (socket says 172.21.15.191:57854) 2023-03-24T23:19:54.488 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.485+0000 7fee36089700 1 -- 172.21.15.191:0/2957420018 learned_addr learned my addr 172.21.15.191:0/2957420018 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:54.488 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee35087700 1 --2- 172.21.15.191:0/2957420018 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee30070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.488 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee36089700 1 -- 172.21.15.191:0/2957420018 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 msgr2=0x7fee30070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:54.488 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee36089700 1 --2- 172.21.15.191:0/2957420018 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee30070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.488 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee36089700 1 -- 172.21.15.191:0/2957420018 >> v1:172.21.15.191:6789/0 conn(0x7fee300cf7e0 legacy=0x7fee300cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.489 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee34886700 1 -- 172.21.15.191:0/2957420018 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048226873 0 0) 0x7fee30160610 con 0x7fee300cf7e0 2023-03-24T23:19:54.489 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee36089700 1 -- 172.21.15.191:0/2957420018 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee301608f0 con 0x7fee30071520 2023-03-24T23:19:54.489 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee35087700 1 --2- 172.21.15.191:0/2957420018 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee30070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-24T23:19:54.489 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee36089700 1 --2- 172.21.15.191:0/2957420018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee30071520 0x7fee3015d370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee28009e50 tx=0x7fee2800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66c2ac7152afad5 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.489 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee34886700 1 -- 172.21.15.191:0/2957420018 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2800e070 con 0x7fee30071520 2023-03-24T23:19:54.489 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee34886700 1 -- 172.21.15.191:0/2957420018 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee2800b9c0 con 0x7fee30071520 2023-03-24T23:19:54.489 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee34886700 1 -- 172.21.15.191:0/2957420018 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2800bd70 con 0x7fee30071520 2023-03-24T23:19:54.490 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee37aec700 1 -- 172.21.15.191:0/2957420018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee30071520 msgr2=0x7fee3015d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.490 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee37aec700 1 --2- 172.21.15.191:0/2957420018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee30071520 0x7fee3015d370 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fee28009e50 tx=0x7fee2800b330 comp rx=0 tx=0).stop 2023-03-24T23:19:54.490 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee37aec700 1 -- 172.21.15.191:0/2957420018 shutdown_connections 2023-03-24T23:19:54.490 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee37aec700 1 --2- 172.21.15.191:0/2957420018 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee30071520 0x7fee3015d370 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.490 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee37aec700 1 --2- 172.21.15.191:0/2957420018 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee30070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.490 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.486+0000 7fee37aec700 1 -- 172.21.15.191:0/2957420018 >> 172.21.15.191:0/2957420018 conn(0x7fee300d0b50 msgr2=0x7fee300cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:54.490 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.487+0000 7fee37aec700 1 -- 172.21.15.191:0/2957420018 shutdown_connections 2023-03-24T23:19:54.491 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.487+0000 7fee37aec700 1 -- 172.21.15.191:0/2957420018 wait complete. 2023-03-24T23:19:54.493 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.488+0000 7fee37aec700 1 Processor -- start 2023-03-24T23:19:54.493 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.488+0000 7fee37aec700 1 -- start start 2023-03-24T23:19:54.493 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.488+0000 7fee37aec700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee30071520 0x7fee301f36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.493 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.488+0000 7fee37aec700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee301482e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.493 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.488+0000 7fee37aec700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee300cf7e0 0x7fee3015e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.494 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.488+0000 7fee37aec700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee301f3280 con 0x7fee300cf7e0 2023-03-24T23:19:54.494 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.488+0000 7fee37aec700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee301f29a0 con 0x7fee30071520 2023-03-24T23:19:54.494 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.488+0000 7fee37aec700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee301f3be0 con 0x7fee300ced00 2023-03-24T23:19:54.494 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee300cf7e0 0x7fee3015e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.494 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee300cf7e0 0x7fee3015e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50464/0 (socket says 172.21.15.191:50464) 2023-03-24T23:19:54.494 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 -- 172.21.15.191:0/2791975716 learned_addr learned my addr 172.21.15.191:0/2791975716 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:54.494 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee35087700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee301482e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.494 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee35888700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee30071520 0x7fee301f36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.495 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 -- 172.21.15.191:0/2791975716 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 msgr2=0x7fee301482e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.495 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee301482e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.495 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 -- 172.21.15.191:0/2791975716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee30071520 msgr2=0x7fee301f36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.495 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee30071520 0x7fee301f36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.495 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 -- 172.21.15.191:0/2791975716 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee28009b30 con 0x7fee300cf7e0 2023-03-24T23:19:54.495 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee36089700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee300cf7e0 0x7fee3015e4c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee280042a0 tx=0x7fee28004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.495 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee267fc700 1 -- 172.21.15.191:0/2791975716 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee28004890 con 0x7fee300cf7e0 2023-03-24T23:19:54.496 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee267fc700 1 -- 172.21.15.191:0/2791975716 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee280099b0 con 0x7fee300cf7e0 2023-03-24T23:19:54.496 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee30160eb0 con 0x7fee300cf7e0 2023-03-24T23:19:54.496 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee267fc700 1 -- 172.21.15.191:0/2791975716 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee2801c680 con 0x7fee300cf7e0 2023-03-24T23:19:54.496 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee30160bd0 con 0x7fee300cf7e0 2023-03-24T23:19:54.496 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee30161190 con 0x7fee300cf7e0 2023-03-24T23:19:54.496 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.489+0000 7fee267fc700 1 -- 172.21.15.191:0/2791975716 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7fee2801c7e0 con 0x7fee300cf7e0 2023-03-24T23:19:54.496 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.490+0000 7fee267fc700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fee14069de0 0x7fee1406c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.497 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.490+0000 7fee267fc700 1 -- 172.21.15.191:0/2791975716 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fee280897f0 con 0x7fee300cf7e0 2023-03-24T23:19:54.497 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.490+0000 7fee35888700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fee14069de0 0x7fee1406c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.497 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.490+0000 7fee35888700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fee14069de0 0x7fee1406c290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee20000f60 tx=0x7fee20007e70 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.497 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.494+0000 7fee267fc700 1 -- 172.21.15.191:0/2791975716 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7fee28055480 con 0x7fee300cf7e0 2023-03-24T23:19:54.680 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.679+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fee301f33c0 con 0x7fee300cf7e0 2023-03-24T23:19:54.681 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.679+0000 7fee267fc700 1 -- 172.21.15.191:0/2791975716 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v18) v1 ==== 93+0+10 (secure 0 0 0) 0x7fee301f33c0 con 0x7fee300cf7e0 2023-03-24T23:19:54.681 INFO:teuthology.orchestra.run.smithi191.stdout:pg_num: 1 2023-03-24T23:19:54.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fee14069de0 msgr2=0x7fee1406c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fee14069de0 0x7fee1406c290 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fee20000f60 tx=0x7fee20007e70 comp rx=0 tx=0).stop 2023-03-24T23:19:54.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee300cf7e0 msgr2=0x7fee3015e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee300cf7e0 0x7fee3015e4c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fee280042a0 tx=0x7fee28004ad0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.683 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 shutdown_connections 2023-03-24T23:19:54.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fee14069de0 0x7fee1406c290 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fee300cf7e0 0x7fee3015e4c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fee30071520 0x7fee301f36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 --2- 172.21.15.191:0/2791975716 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fee300ced00 0x7fee301482e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 >> 172.21.15.191:0/2791975716 conn(0x7fee300d0b50 msgr2=0x7fee30156930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:54.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 shutdown_connections 2023-03-24T23:19:54.684 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.681+0000 7fee37aec700 1 -- 172.21.15.191:0/2791975716 wait complete. 2023-03-24T23:19:54.694 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-24T23:19:54.816 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.815+0000 7f1ffbcc1700 1 Processor -- start 2023-03-24T23:19:54.816 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.815+0000 7f1ffbcc1700 1 -- start start 2023-03-24T23:19:54.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffbcc1700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff4070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffbcc1700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1ff4071520 0x7f1ff415d370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffbcc1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ff4160610 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffbcc1700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ff40cf170 con 0x7f1ff4071520 2023-03-24T23:19:54.818 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffbcc1700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40719a0 con 0x7f1ff40ced00 2023-03-24T23:19:54.819 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffa25e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1ff4071520 0x7f1ff415d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.819 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffa25e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f1ff4071520 0x7f1ff415d370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50472/0 (socket says 172.21.15.191:50472) 2023-03-24T23:19:54.819 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffa25e700 1 -- 172.21.15.191:0/4179806773 learned_addr learned my addr 172.21.15.191:0/4179806773 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:54.819 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ff9a5d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f1ff40cf7e0 0x7f1ff40cbd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57868/0 (socket says 172.21.15.191:57868) 2023-03-24T23:19:54.819 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ff925c700 1 --2- 172.21.15.191:0/4179806773 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff4070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.819 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffa25e700 1 -- 172.21.15.191:0/4179806773 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 msgr2=0x7f1ff4070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:54.819 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffa25e700 1 --2- 172.21.15.191:0/4179806773 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff4070fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.819 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffa25e700 1 -- 172.21.15.191:0/4179806773 >> v1:172.21.15.191:6789/0 conn(0x7f1ff40cf7e0 legacy=0x7f1ff40cbd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.820 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffa25e700 1 -- 172.21.15.191:0/4179806773 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ff41608f0 con 0x7f1ff4071520 2023-03-24T23:19:54.820 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ff8a5b700 1 -- 172.21.15.191:0/4179806773 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 113956429 0 0) 0x7f1ff4160610 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.820 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ffa25e700 1 --2- 172.21.15.191:0/4179806773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff4071520 0x7f1ff415d370 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009e50 tx=0x7f1fec00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=320d4e6446f6cd31 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.820 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ff8a5b700 1 -- 172.21.15.191:0/4179806773 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00e070 con 0x7f1ff4071520 2023-03-24T23:19:54.820 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ff8a5b700 1 -- 172.21.15.191:0/4179806773 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec00b9c0 con 0x7f1ff4071520 2023-03-24T23:19:54.820 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.816+0000 7f1ff8a5b700 1 -- 172.21.15.191:0/4179806773 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec00bd70 con 0x7f1ff4071520 2023-03-24T23:19:54.824 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.817+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/4179806773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff4071520 msgr2=0x7f1ff415d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.824 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.817+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/4179806773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff4071520 0x7f1ff415d370 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1fec009e50 tx=0x7f1fec00b330 comp rx=0 tx=0).stop 2023-03-24T23:19:54.824 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.817+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/4179806773 shutdown_connections 2023-03-24T23:19:54.824 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.817+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/4179806773 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff4071520 0x7f1ff415d370 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.824 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.817+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/4179806773 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff4070fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.825 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.817+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/4179806773 >> 172.21.15.191:0/4179806773 conn(0x7f1ff40d0b50 msgr2=0x7f1ff40cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:54.825 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.817+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/4179806773 shutdown_connections 2023-03-24T23:19:54.825 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.817+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/4179806773 wait complete. 2023-03-24T23:19:54.825 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.818+0000 7f1ffbcc1700 1 Processor -- start 2023-03-24T23:19:54.825 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.818+0000 7f1ffbcc1700 1 -- start start 2023-03-24T23:19:54.825 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffbcc1700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ff4071520 0x7f1ff40c6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.825 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffbcc1700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff415ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.826 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffbcc1700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff40cf7e0 0x7f1ff415ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.826 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffbcc1700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4152eb0 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.826 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffbcc1700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff4150e90 con 0x7f1ff4071520 2023-03-24T23:19:54.826 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffbcc1700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ff40c3b00 con 0x7f1ff40ced00 2023-03-24T23:19:54.826 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff40cf7e0 0x7f1ff415ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.826 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff40cf7e0 0x7f1ff415ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50478/0 (socket says 172.21.15.191:50478) 2023-03-24T23:19:54.826 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 -- 172.21.15.191:0/2917105809 learned_addr learned my addr 172.21.15.191:0/2917105809 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ff925c700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff415ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ff9a5d700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ff4071520 0x7f1ff40c6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 -- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 msgr2=0x7f1ff415ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff415ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 -- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ff4071520 msgr2=0x7f1ff40c6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ff4071520 0x7f1ff40c6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 -- 172.21.15.191:0/2917105809 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fec009b30 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.827 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ff925c700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff415ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:54.828 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ff9a5d700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ff4071520 0x7f1ff40c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:54.828 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.819+0000 7f1ffa25e700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff40cf7e0 0x7f1ff415ef80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40cd140 tx=0x7f1fec004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.828 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.820+0000 7f1fea7fc700 1 -- 172.21.15.191:0/2917105809 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec003ea0 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.828 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.820+0000 7f1fea7fc700 1 -- 172.21.15.191:0/2917105809 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fec0099b0 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.828 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.820+0000 7f1fea7fc700 1 -- 172.21.15.191:0/2917105809 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fec01c680 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.828 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.820+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ff4160eb0 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.828 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.820+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ff4160bd0 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.829 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.821+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ff4161190 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.829 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.821+0000 7f1fea7fc700 1 -- 172.21.15.191:0/2917105809 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f1fec01c7e0 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.829 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.821+0000 7f1fea7fc700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1fd8069de0 0x7f1fd806c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:54.829 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.822+0000 7f1ff9a5d700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1fd8069de0 0x7f1fd806c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:54.829 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.822+0000 7f1fea7fc700 1 -- 172.21.15.191:0/2917105809 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1fec0899a0 con 0x7f1ff40cf7e0 2023-03-24T23:19:54.829 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.822+0000 7f1ff9a5d700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1fd8069de0 0x7f1fd806c290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4007fd0 tx=0x7f1fe4009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:54.830 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:54.826+0000 7f1fea7fc700 1 -- 172.21.15.191:0/2917105809 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f1fec055630 con 0x7f1ff40cf7e0 2023-03-24T23:19:55.015 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.014+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f1ff40ceae0 con 0x7f1ff40cf7e0 2023-03-24T23:19:55.015 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.014+0000 7f1fea7fc700 1 -- 172.21.15.191:0/2917105809 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1fec058c50 con 0x7f1ff40cf7e0 2023-03-24T23:19:55.016 INFO:teuthology.orchestra.run.smithi191.stdout:pg_num: 8 2023-03-24T23:19:55.017 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1fd8069de0 msgr2=0x7f1fd806c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1fd8069de0 0x7f1fd806c290 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1fe4007fd0 tx=0x7f1fe4009040 comp rx=0 tx=0).stop 2023-03-24T23:19:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff40cf7e0 msgr2=0x7f1ff415ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff40cf7e0 0x7f1ff415ef80 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1ff40cd140 tx=0x7f1fec004ad0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 shutdown_connections 2023-03-24T23:19:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1fd8069de0 0x7f1fd806c290 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1ff40cf7e0 0x7f1ff415ef80 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.018 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ff4071520 0x7f1ff40c6230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.019 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 --2- 172.21.15.191:0/2917105809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ff40ced00 0x7f1ff415ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.019 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 >> 172.21.15.191:0/2917105809 conn(0x7f1ff40d0b50 msgr2=0x7f1ff40b3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:55.019 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 shutdown_connections 2023-03-24T23:19:55.019 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.016+0000 7f1ffbcc1700 1 -- 172.21.15.191:0/2917105809 wait complete. 2023-03-24T23:19:55.029 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-03-24T23:19:55.151 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.150+0000 7f139ce0a700 1 Processor -- start 2023-03-24T23:19:55.151 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.150+0000 7f139ce0a700 1 -- start start 2023-03-24T23:19:55.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.150+0000 7f139ce0a700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f13980cbdb0 0x7f139815de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.150+0000 7f139ce0a700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f13980cf080 0x7f1398159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.153 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.150+0000 7f139ce0a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139815d990 con 0x7f139815a400 2023-03-24T23:19:55.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.150+0000 7f139ce0a700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139815aea0 con 0x7f13980cf080 2023-03-24T23:19:55.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.150+0000 7f139ce0a700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139815a8e0 con 0x7f13980cbdb0 2023-03-24T23:19:55.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f1395d9b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f13980cf080 0x7f1398159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f1396d9d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f139815a400 0x7f1398071050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57888/0 (socket says 172.21.15.191:57888) 2023-03-24T23:19:55.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f1396d9d700 1 -- 172.21.15.191:0/1359838135 learned_addr learned my addr 172.21.15.191:0/1359838135 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:55.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f139659c700 1 --2- 172.21.15.191:0/1359838135 >> v2:172.21.15.196:3300/0 conn(0x7f13980cbdb0 0x7f139815de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.154 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f1395d9b700 1 -- 172.21.15.191:0/1359838135 >> v2:172.21.15.196:3300/0 conn(0x7f13980cbdb0 msgr2=0x7f139815de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:55.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f1395d9b700 1 --2- 172.21.15.191:0/1359838135 >> v2:172.21.15.196:3300/0 conn(0x7f13980cbdb0 0x7f139815de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f1395d9b700 1 -- 172.21.15.191:0/1359838135 >> v1:172.21.15.191:6789/0 conn(0x7f139815a400 legacy=0x7f1398071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f139559a700 1 -- 172.21.15.191:0/1359838135 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204172870 0 0) 0x7f139815d990 con 0x7f139815a400 2023-03-24T23:19:55.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f1395d9b700 1 -- 172.21.15.191:0/1359838135 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139815abc0 con 0x7f13980cf080 2023-03-24T23:19:55.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f139659c700 1 --2- 172.21.15.191:0/1359838135 >> v2:172.21.15.196:3300/0 conn(0x7f13980cbdb0 0x7f139815de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:55.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f1395d9b700 1 --2- 172.21.15.191:0/1359838135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cf080 0x7f1398159ec0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f13980d0ff0 tx=0x7f138c009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7c28d942cb3bab4 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f139559a700 1 -- 172.21.15.191:0/1359838135 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c011830 con 0x7f13980cf080 2023-03-24T23:19:55.155 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f139559a700 1 -- 172.21.15.191:0/1359838135 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138c011990 con 0x7f13980cf080 2023-03-24T23:19:55.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.151+0000 7f139559a700 1 -- 172.21.15.191:0/1359838135 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f138c011d20 con 0x7f13980cf080 2023-03-24T23:19:55.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.152+0000 7f139ce0a700 1 -- 172.21.15.191:0/1359838135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cf080 msgr2=0x7f1398159ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.152+0000 7f139ce0a700 1 --2- 172.21.15.191:0/1359838135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cf080 0x7f1398159ec0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f13980d0ff0 tx=0x7f138c009b20 comp rx=0 tx=0).stop 2023-03-24T23:19:55.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.152+0000 7f139ce0a700 1 -- 172.21.15.191:0/1359838135 shutdown_connections 2023-03-24T23:19:55.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.152+0000 7f139ce0a700 1 --2- 172.21.15.191:0/1359838135 >> v2:172.21.15.196:3300/0 conn(0x7f13980cbdb0 0x7f139815de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.152+0000 7f139ce0a700 1 --2- 172.21.15.191:0/1359838135 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cf080 0x7f1398159ec0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.156 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.152+0000 7f139ce0a700 1 -- 172.21.15.191:0/1359838135 >> 172.21.15.191:0/1359838135 conn(0x7f13980cfef0 msgr2=0x7f13980d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:55.157 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.152+0000 7f139ce0a700 1 -- 172.21.15.191:0/1359838135 shutdown_connections 2023-03-24T23:19:55.157 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.152+0000 7f139ce0a700 1 -- 172.21.15.191:0/1359838135 wait complete. 2023-03-24T23:19:55.158 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.153+0000 7f139ce0a700 1 Processor -- start 2023-03-24T23:19:55.159 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.153+0000 7f139ce0a700 1 -- start start 2023-03-24T23:19:55.159 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139ce0a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cbdb0 0x7f13980c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.159 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139ce0a700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13980cf080 0x7f1398149950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.159 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139ce0a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f139815a400 0x7f13981580d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.159 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139ce0a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13980c4990 con 0x7f13980cbdb0 2023-03-24T23:19:55.159 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139ce0a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13980c7860 con 0x7f139815a400 2023-03-24T23:19:55.160 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139ce0a700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13980c6520 con 0x7f13980cf080 2023-03-24T23:19:55.160 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cbdb0 0x7f13980c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.160 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cbdb0 0x7f13980c5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50490/0 (socket says 172.21.15.191:50490) 2023-03-24T23:19:55.160 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 -- 172.21.15.191:0/755359084 learned_addr learned my addr 172.21.15.191:0/755359084 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:55.160 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 -- 172.21.15.191:0/755359084 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13980cf080 msgr2=0x7f1398149950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.160 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13980cf080 0x7f1398149950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.160 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 -- 172.21.15.191:0/755359084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f139815a400 msgr2=0x7f13981580d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.160 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f139815a400 0x7f13981580d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 -- 172.21.15.191:0/755359084 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138c0094a0 con 0x7f13980cbdb0 2023-03-24T23:19:55.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f139659c700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cbdb0 0x7f13980c5fe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f138800c7f0 tx=0x7f138800cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.154+0000 7f1386ffd700 1 -- 172.21.15.191:0/755359084 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388011800 con 0x7f13980cbdb0 2023-03-24T23:19:55.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.155+0000 7f1386ffd700 1 -- 172.21.15.191:0/755359084 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1388011960 con 0x7f13980cbdb0 2023-03-24T23:19:55.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.155+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13980cc180 con 0x7f13980cbdb0 2023-03-24T23:19:55.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.155+0000 7f1386ffd700 1 -- 172.21.15.191:0/755359084 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1388011d10 con 0x7f13980cbdb0 2023-03-24T23:19:55.161 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.155+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f139815d620 con 0x7f13980cbdb0 2023-03-24T23:19:55.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.156+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f139815e360 con 0x7f13980cbdb0 2023-03-24T23:19:55.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.156+0000 7f1386ffd700 1 -- 172.21.15.191:0/755359084 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f1388018360 con 0x7f13980cbdb0 2023-03-24T23:19:55.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.156+0000 7f1386ffd700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1378069e10 0x7f137806c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.156+0000 7f1395d9b700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1378069e10 0x7f137806c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.156+0000 7f1386ffd700 1 -- 172.21.15.191:0/755359084 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1388087c20 con 0x7f13980cbdb0 2023-03-24T23:19:55.162 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.157+0000 7f1395d9b700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1378069e10 0x7f137806c2c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f13980cf450 tx=0x7f138c010000 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.163 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.161+0000 7f1386ffd700 1 -- 172.21.15.191:0/755359084 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f1388057260 con 0x7f13980cbdb0 2023-03-24T23:19:55.389 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.388+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1398074580 con 0x7f13980cbdb0 2023-03-24T23:19:55.390 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.389+0000 7f1386ffd700 1 -- 172.21.15.191:0/755359084 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3536 (secure 0 0 0) 0x7f138801fde0 con 0x7f13980cbdb0 2023-03-24T23:19:55.391 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:55.391 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4235,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6836/4143607744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6834","nonce":4143607744},{"type":"v1","addr":"172.21.15.191:6836","nonce":4143607744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4244,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6834/2625325997","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":2625325997},{"type":"v1","addr":"172.21.15.196:6834","nonce":2625325997}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.191:6837/1726689837","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6835","nonce":1726689837},{"type":"v1","addr":"172.21.15.191:6837","nonce":1726689837}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":2},{"gid":4288,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/1699153396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6833","nonce":1699153396},{"type":"v1","addr":"172.21.15.196:6835","nonce":1699153396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":3}],"filesystems":[]} 2023-03-24T23:19:55.392 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1378069e10 msgr2=0x7f137806c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1378069e10 0x7f137806c2c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f13980cf450 tx=0x7f138c010000 comp rx=0 tx=0).stop 2023-03-24T23:19:55.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cbdb0 msgr2=0x7f13980c5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cbdb0 0x7f13980c5fe0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f138800c7f0 tx=0x7f138800cbb0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 shutdown_connections 2023-03-24T23:19:55.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f1378069e10 0x7f137806c2c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.393 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f13980cbdb0 0x7f13980c5fe0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f139815a400 0x7f13981580d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 --2- 172.21.15.191:0/755359084 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f13980cf080 0x7f1398149950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 >> 172.21.15.191:0/755359084 conn(0x7f13980cfef0 msgr2=0x7f1398070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:55.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 shutdown_connections 2023-03-24T23:19:55.394 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.391+0000 7f139ce0a700 1 -- 172.21.15.191:0/755359084 wait complete. 2023-03-24T23:19:55.395 INFO:teuthology.orchestra.run.smithi191.stderr:dumped fsmap epoch 3 2023-03-24T23:19:55.404 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-03-24T23:19:55.524 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.523+0000 7feedfb1b700 1 Processor -- start 2023-03-24T23:19:55.525 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.523+0000 7feedfb1b700 1 -- start start 2023-03-24T23:19:55.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedfb1b700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7feed81602c0 0x7feed8071050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedfb1b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feed8071590 0x7feed80cdac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedfb1b700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed80719e0 con 0x7feed80ce000 2023-03-24T23:19:55.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedfb1b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed80b4590 con 0x7feed8071590 2023-03-24T23:19:55.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedfb1b700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feed80ce4a0 con 0x7feed81602c0 2023-03-24T23:19:55.527 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedd0b6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feed8071590 0x7feed80cdac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feede0b8700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7feed80ce000 0x7feed80ca540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57898/0 (socket says 172.21.15.191:57898) 2023-03-24T23:19:55.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedd0b6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7feed8071590 0x7feed80cdac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50498/0 (socket says 172.21.15.191:50498) 2023-03-24T23:19:55.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feede0b8700 1 -- 172.21.15.191:0/3472651114 learned_addr learned my addr 172.21.15.191:0/3472651114 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:55.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedd8b7700 1 --2- 172.21.15.191:0/3472651114 >> v2:172.21.15.196:3300/0 conn(0x7feed81602c0 0x7feed8071050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedd0b6700 1 -- 172.21.15.191:0/3472651114 >> v2:172.21.15.196:3300/0 conn(0x7feed81602c0 msgr2=0x7feed8071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:55.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedd0b6700 1 --2- 172.21.15.191:0/3472651114 >> v2:172.21.15.196:3300/0 conn(0x7feed81602c0 0x7feed8071050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedd0b6700 1 -- 172.21.15.191:0/3472651114 >> v1:172.21.15.191:6789/0 conn(0x7feed80ce000 legacy=0x7feed80ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.528 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedd0b6700 1 -- 172.21.15.191:0/3472651114 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed80cf140 con 0x7feed8071590 2023-03-24T23:19:55.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.524+0000 7feedc8b5700 1 -- 172.21.15.191:0/3472651114 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161740708 0 0) 0x7feed0001490 con 0x7feed80ce000 2023-03-24T23:19:55.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedd0b6700 1 --2- 172.21.15.191:0/3472651114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed8071590 0x7feed80cdac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feed8070e50 tx=0x7feed4009b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2d49368f8e52292 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedc8b5700 1 -- 172.21.15.191:0/3472651114 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4011830 con 0x7feed8071590 2023-03-24T23:19:55.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedc8b5700 1 -- 172.21.15.191:0/3472651114 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed4011990 con 0x7feed8071590 2023-03-24T23:19:55.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedc8b5700 1 -- 172.21.15.191:0/3472651114 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4011d20 con 0x7feed8071590 2023-03-24T23:19:55.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedfb1b700 1 -- 172.21.15.191:0/3472651114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed8071590 msgr2=0x7feed80cdac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.529 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3472651114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed8071590 0x7feed80cdac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7feed8070e50 tx=0x7feed4009b20 comp rx=0 tx=0).stop 2023-03-24T23:19:55.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedfb1b700 1 -- 172.21.15.191:0/3472651114 shutdown_connections 2023-03-24T23:19:55.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3472651114 >> v2:172.21.15.196:3300/0 conn(0x7feed81602c0 0x7feed8071050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3472651114 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed8071590 0x7feed80cdac0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedfb1b700 1 -- 172.21.15.191:0/3472651114 >> 172.21.15.191:0/3472651114 conn(0x7feed80cfef0 msgr2=0x7feed80d02c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:55.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedfb1b700 1 -- 172.21.15.191:0/3472651114 shutdown_connections 2023-03-24T23:19:55.530 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.525+0000 7feedfb1b700 1 -- 172.21.15.191:0/3472651114 wait complete. 2023-03-24T23:19:55.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.526+0000 7feedfb1b700 1 Processor -- start 2023-03-24T23:19:55.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedfb1b700 1 -- start start 2023-03-24T23:19:55.532 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedfb1b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feed8071590 0x7feed8158380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedfb1b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed80ce000 0x7feed81588c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedfb1b700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feed81602c0 0x7feed80c3ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedfb1b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed815a210 con 0x7feed80ce000 2023-03-24T23:19:55.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedfb1b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed8159170 con 0x7feed8071590 2023-03-24T23:19:55.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedfb1b700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feed8159450 con 0x7feed81602c0 2023-03-24T23:19:55.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedd0b6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed80ce000 0x7feed81588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.533 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedd0b6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed80ce000 0x7feed81588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50506/0 (socket says 172.21.15.191:50506) 2023-03-24T23:19:55.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.527+0000 7feedd0b6700 1 -- 172.21.15.191:0/3335584389 learned_addr learned my addr 172.21.15.191:0/3335584389 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:55.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedd0b6700 1 -- 172.21.15.191:0/3335584389 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feed81602c0 msgr2=0x7feed80c3ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedd0b6700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feed81602c0 0x7feed80c3ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedd0b6700 1 -- 172.21.15.191:0/3335584389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feed8071590 msgr2=0x7feed8158380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedd8b7700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feed8071590 0x7feed8158380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedd0b6700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feed8071590 0x7feed8158380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedd0b6700 1 -- 172.21.15.191:0/3335584389 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed40094a0 con 0x7feed80ce000 2023-03-24T23:19:55.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedd0b6700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed80ce000 0x7feed81588c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feed4009b60 tx=0x7feed4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feece7fc700 1 -- 172.21.15.191:0/3335584389 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed40036a0 con 0x7feed80ce000 2023-03-24T23:19:55.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feece7fc700 1 -- 172.21.15.191:0/3335584389 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feed401f070 con 0x7feed80ce000 2023-03-24T23:19:55.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed80b3ed0 con 0x7feed80ce000 2023-03-24T23:19:55.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feece7fc700 1 -- 172.21.15.191:0/3335584389 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed4003a60 con 0x7feed80ce000 2023-03-24T23:19:55.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.528+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed80ce6a0 con 0x7feed80ce000 2023-03-24T23:19:55.535 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.529+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed80b41b0 con 0x7feed80ce000 2023-03-24T23:19:55.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.529+0000 7feece7fc700 1 -- 172.21.15.191:0/3335584389 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7feed4003bc0 con 0x7feed80ce000 2023-03-24T23:19:55.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.530+0000 7feece7fc700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7feebc069de0 0x7feebc06c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.530+0000 7feedd8b7700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7feebc069de0 0x7feebc06c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.530+0000 7feece7fc700 1 -- 172.21.15.191:0/3335584389 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7feed4021070 con 0x7feed80ce000 2023-03-24T23:19:55.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.530+0000 7feedd8b7700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7feebc069de0 0x7feebc06c290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feec8009770 tx=0x7feec8008040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.534+0000 7feece7fc700 1 -- 172.21.15.191:0/3335584389 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7feed4055420 con 0x7feed80ce000 2023-03-24T23:19:55.725 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.724+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feed8074580 con 0x7feed80ce000 2023-03-24T23:19:55.726 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.725+0000 7feece7fc700 1 -- 172.21.15.191:0/3335584389 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+14242 (secure 0 0 0) 0x7feed4015090 con 0x7feed80ce000 2023-03-24T23:19:55.726 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:55.726 INFO:teuthology.orchestra.run.smithi191.stdout:{"epoch":19,"fsid":"72b355de-1f4b-4f02-899f-839bc265bc0f","created":"2023-03-24T23:19:34.494902+0000","modified":"2023-03-24T23:19:54.990411+0000","last_up_change":"2023-03-24T23:19:45.928145+0000","last_in_change":"2023-03-24T23:19:36.946912+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-03-24T23:19:46.103710+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2023-03-24T23:19:52.893681+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"0ca3720a-fe6e-4b70-a32f-fef43c5da336","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6817","nonce":97650},{"type":"v1","addr":"172.21.15.191:6819","nonce":97650}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6821","nonce":97650},{"type":"v1","addr":"172.21.15.191:6823","nonce":97650}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6829","nonce":97650},{"type":"v1","addr":"172.21.15.191:6831","nonce":97650}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6825","nonce":97650},{"type":"v1","addr":"172.21.15.191:6827","nonce":97650}]},"public_addr":"172.21.15.191:6819/97650","cluster_addr":"172.21.15.191:6823/97650","heartbeat_back_addr":"172.21.15.191:6831/97650","heartbeat_front_addr":"172.21.15.191:6827/97650","state":["exists","up"]},{"osd":1,"uuid":"55f40987-e709-41aa-84f1-7a0b426e984d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6816","nonce":97651},{"type":"v1","addr":"172.21.15.191:6818","nonce":97651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6820","nonce":97651},{"type":"v1","addr":"172.21.15.191:6822","nonce":97651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6828","nonce":97651},{"type":"v1","addr":"172.21.15.191:6830","nonce":97651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6824","nonce":97651},{"type":"v1","addr":"172.21.15.191:6826","nonce":97651}]},"public_addr":"172.21.15.191:6818/97651","cluster_addr":"172.21.15.191:6822/97651","heartbeat_back_addr":"172.21.15.191:6830/97651","heartbeat_front_addr":"172.21.15.191:6826/97651","state":["exists","up"]},{"osd":2,"uuid":"b2e378f2-4cdb-4900-8f17-83affc66d8ce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6808","nonce":97648},{"type":"v1","addr":"172.21.15.191:6809","nonce":97648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6810","nonce":97648},{"type":"v1","addr":"172.21.15.191:6811","nonce":97648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6814","nonce":97648},{"type":"v1","addr":"172.21.15.191:6815","nonce":97648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6812","nonce":97648},{"type":"v1","addr":"172.21.15.191:6813","nonce":97648}]},"public_addr":"172.21.15.191:6809/97648","cluster_addr":"172.21.15.191:6811/97648","heartbeat_back_addr":"172.21.15.191:6815/97648","heartbeat_front_addr":"172.21.15.191:6813/97648","state":["exists","up"]},{"osd":3,"uuid":"b4483b81-cc61-477a-ada6-4620185b9f6b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6800","nonce":97649},{"type":"v1","addr":"172.21.15.191:6801","nonce":97649}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6802","nonce":97649},{"type":"v1","addr":"172.21.15.191:6803","nonce":97649}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6806","nonce":97649},{"type":"v1","addr":"172.21.15.191:6807","nonce":97649}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.191:6804","nonce":97649},{"type":"v1","addr":"172.21.15.191:6805","nonce":97649}]},"public_addr":"172.21.15.191:6801/97649","cluster_addr":"172.21.15.191:6803/97649","heartbeat_back_addr":"172.21.15.191:6807/97649","heartbeat_front_addr":"172.21.15.191:6805/97649","state":["exists","up"]},{"osd":4,"uuid":"f133b7f6-ca18-4ff3-a71f-d9ae1593ce57","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":97900},{"type":"v1","addr":"172.21.15.196:6809","nonce":97900}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":97900},{"type":"v1","addr":"172.21.15.196:6811","nonce":97900}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":97900},{"type":"v1","addr":"172.21.15.196:6815","nonce":97900}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":97900},{"type":"v1","addr":"172.21.15.196:6813","nonce":97900}]},"public_addr":"172.21.15.196:6809/97900","cluster_addr":"172.21.15.196:6811/97900","heartbeat_back_addr":"172.21.15.196:6815/97900","heartbeat_front_addr":"172.21.15.196:6813/97900","state":["exists","up"]},{"osd":5,"uuid":"b284d515-2796-4d33-81a8-0e42195d49bb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":97901},{"type":"v1","addr":"172.21.15.196:6801","nonce":97901}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":97901},{"type":"v1","addr":"172.21.15.196:6803","nonce":97901}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":97901},{"type":"v1","addr":"172.21.15.196:6807","nonce":97901}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":97901},{"type":"v1","addr":"172.21.15.196:6805","nonce":97901}]},"public_addr":"172.21.15.196:6801/97901","cluster_addr":"172.21.15.196:6803/97901","heartbeat_back_addr":"172.21.15.196:6807/97901","heartbeat_front_addr":"172.21.15.196:6805/97901","state":["exists","up"]},{"osd":6,"uuid":"37ee78d6-7798-4d17-ad11-4fffd8355d73","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":97902},{"type":"v1","addr":"172.21.15.196:6817","nonce":97902}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":97902},{"type":"v1","addr":"172.21.15.196:6819","nonce":97902}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":97902},{"type":"v1","addr":"172.21.15.196:6823","nonce":97902}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":97902},{"type":"v1","addr":"172.21.15.196:6821","nonce":97902}]},"public_addr":"172.21.15.196:6817/97902","cluster_addr":"172.21.15.196:6819/97902","heartbeat_back_addr":"172.21.15.196:6823/97902","heartbeat_front_addr":"172.21.15.196:6821/97902","state":["exists","up"]},{"osd":7,"uuid":"0c1c2708-0d59-4f5b-89aa-75ea67df828e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6824","nonce":97907},{"type":"v1","addr":"172.21.15.196:6825","nonce":97907}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":97907},{"type":"v1","addr":"172.21.15.196:6827","nonce":97907}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":97907},{"type":"v1","addr":"172.21.15.196:6831","nonce":97907}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":97907},{"type":"v1","addr":"172.21.15.196:6829","nonce":97907}]},"public_addr":"172.21.15.196:6825/97907","cluster_addr":"172.21.15.196:6827/97907","heartbeat_back_addr":"172.21.15.196:6831/97907","heartbeat_front_addr":"172.21.15.196:6829/97907","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.195233+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.178597+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.205385+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.186833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.213154+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.109613+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.150885+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-03-24T23:19:42.215448+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-03-24T23:19:55.728 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7feebc069de0 msgr2=0x7feebc06c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.728 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7feebc069de0 0x7feebc06c290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7feec8009770 tx=0x7feec8008040 comp rx=0 tx=0).stop 2023-03-24T23:19:55.729 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed80ce000 msgr2=0x7feed81588c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.729 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed80ce000 0x7feed81588c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7feed4009b60 tx=0x7feed4004750 comp rx=0 tx=0).stop 2023-03-24T23:19:55.729 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 shutdown_connections 2023-03-24T23:19:55.729 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7feebc069de0 0x7feebc06c290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.729 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feed81602c0 0x7feed80c3ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.729 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feed8071590 0x7feed8158380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.729 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 --2- 172.21.15.191:0/3335584389 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7feed80ce000 0x7feed81588c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.730 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 >> 172.21.15.191:0/3335584389 conn(0x7feed80cfef0 msgr2=0x7feed80b19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:55.730 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 shutdown_connections 2023-03-24T23:19:55.730 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.727+0000 7feedfb1b700 1 -- 172.21.15.191:0/3335584389 wait complete. 2023-03-24T23:19:55.740 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-03-24T23:19:46.103710+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2023-03-24T23:19:52.893681+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 4.980000019073486, 'score_stable': 4.980000019073486, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 4.380000114440918, 'raw_score_stable': 4.380000114440918, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-03-24T23:19:55.741 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-03-24T23:19:55.862 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.861+0000 7f195043a700 1 Processor -- start 2023-03-24T23:19:55.863 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.861+0000 7f195043a700 1 -- start start 2023-03-24T23:19:55.865 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f195043a700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f194815cfd0 0x7f19480cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.865 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f195043a700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480716f0 0x7f19480ce310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.865 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f195043a700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f194815d470 con 0x7f19480cda00 2023-03-24T23:19:55.865 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f195043a700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f194815e080 con 0x7f194815cfd0 2023-03-24T23:19:55.865 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f195043a700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19480cfd10 con 0x7f19480716f0 2023-03-24T23:19:55.865 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f194e1d6700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f19480cda00 0x7f1948070fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57912/0 (socket says 172.21.15.191:57912) 2023-03-24T23:19:55.866 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f194e1d6700 1 -- 172.21.15.191:0/3605308950 learned_addr learned my addr 172.21.15.191:0/3605308950 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:55.866 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f194d9d5700 1 --2- 172.21.15.191:0/3605308950 >> v2:172.21.15.196:3300/0 conn(0x7f194815cfd0 0x7f19480cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.866 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f194e9d7700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480716f0 0x7f19480ce310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.866 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879967695 0 0) 0x7f194815d470 con 0x7f19480cda00 2023-03-24T23:19:55.866 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.862+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19340035b0 con 0x7f19480cda00 2023-03-24T23:19:55.866 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 <== mon.0 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3579997387 0 0) 0x7f19340035b0 con 0x7f19480cda00 2023-03-24T23:19:55.866 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480716f0 msgr2=0x7f19480ce310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:55.866 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480716f0 0x7f19480ce310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.867 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 >> v2:172.21.15.196:3300/0 conn(0x7f194815cfd0 msgr2=0x7f19480cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:55.867 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 --2- 172.21.15.191:0/3605308950 >> v2:172.21.15.196:3300/0 conn(0x7f194815cfd0 0x7f19480cddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.867 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194815d750 con 0x7f19480cda00 2023-03-24T23:19:55.867 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 <== mon.0 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1537907258 0 0) 0x7f1938001670 con 0x7f19480cda00 2023-03-24T23:19:55.867 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 >> v1:172.21.15.191:6789/0 conn(0x7f19480cda00 legacy=0x7f1948070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.867 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934003a00 0x7f1934003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.867 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480716f0 0x7f1934004850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.868 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1934004d90 0x7f1934005200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.868 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194815e080 con 0x7f1934004d90 2023-03-24T23:19:55.868 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19480cfd10 con 0x7f1934003a00 2023-03-24T23:19:55.868 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f193400ba10 con 0x7f19480716f0 2023-03-24T23:19:55.868 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194e9d7700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1934004d90 0x7f1934005200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.868 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d9d5700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480716f0 0x7f1934004850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.868 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194e1d6700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934003a00 0x7f1934003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.868 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194e9d7700 1 -- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480716f0 msgr2=0x7f1934004850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:55.869 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194e9d7700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480716f0 0x7f1934004850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.869 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194e9d7700 1 -- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934003a00 msgr2=0x7f1934003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:55.869 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194e9d7700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934003a00 0x7f1934003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.869 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194e9d7700 1 -- 172.21.15.191:0/3605308950 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19340035b0 con 0x7f1934004d90 2023-03-24T23:19:55.869 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194e9d7700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1934004d90 0x7f1934005200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1940009f90 tx=0x7f1934005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59afecd1730b0ba7 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.869 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194000ebd0 con 0x7f1934004d90 2023-03-24T23:19:55.869 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194000f040 con 0x7f1934004d90 2023-03-24T23:19:55.870 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.863+0000 7f194d1d4700 1 -- 172.21.15.191:0/3605308950 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1940014480 con 0x7f1934004d90 2023-03-24T23:19:55.872 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.864+0000 7f195043a700 1 -- 172.21.15.191:0/3605308950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1934004d90 msgr2=0x7f1934005200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.872 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.864+0000 7f195043a700 1 --2- 172.21.15.191:0/3605308950 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f1934004d90 0x7f1934005200 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f1940009f90 tx=0x7f1934005880 comp rx=0 tx=0).stop 2023-03-24T23:19:55.872 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.864+0000 7f194e9d7700 1 -- 172.21.15.191:0/3605308950 reap_dead start 2023-03-24T23:19:55.872 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.864+0000 7f195043a700 1 -- 172.21.15.191:0/3605308950 shutdown_connections 2023-03-24T23:19:55.872 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.864+0000 7f195043a700 1 -- 172.21.15.191:0/3605308950 >> 172.21.15.191:0/3605308950 conn(0x7f19480d0b50 msgr2=0x7f19480cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:55.873 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.864+0000 7f195043a700 1 -- 172.21.15.191:0/3605308950 shutdown_connections 2023-03-24T23:19:55.873 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.864+0000 7f195043a700 1 -- 172.21.15.191:0/3605308950 wait complete. 2023-03-24T23:19:55.873 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.865+0000 7f195043a700 1 Processor -- start 2023-03-24T23:19:55.873 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.865+0000 7f195043a700 1 -- start start 2023-03-24T23:19:55.873 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f195043a700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934004d90 0x7f194815f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.873 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f195043a700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19480c69d0 0x7f19480c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.873 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f195043a700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480c4a70 0x7f1948144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f195043a700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1948154fd0 con 0x7f19480c69d0 2023-03-24T23:19:55.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f195043a700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f194815f970 con 0x7f1934004d90 2023-03-24T23:19:55.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f195043a700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f194815fc50 con 0x7f19480c4a70 2023-03-24T23:19:55.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19480c69d0 0x7f19480c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19480c69d0 0x7f19480c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50526/0 (socket says 172.21.15.191:50526) 2023-03-24T23:19:55.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 -- 172.21.15.191:0/1000144118 learned_addr learned my addr 172.21.15.191:0/1000144118 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:55.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194e9d7700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480c4a70 0x7f1948144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 -- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480c4a70 msgr2=0x7f1948144da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194e1d6700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934004d90 0x7f194815f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480c4a70 0x7f1948144da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 -- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934004d90 msgr2=0x7f194815f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:55.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934004d90 0x7f194815f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:55.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 -- 172.21.15.191:0/1000144118 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f194815dd10 con 0x7f19480c69d0 2023-03-24T23:19:55.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194e9d7700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480c4a70 0x7f1948144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:55.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194e1d6700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934004d90 0x7f194815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:55.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f194d9d5700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19480c69d0 0x7f19480c4530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1944011aa0 tx=0x7f1944011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f193effd700 1 -- 172.21.15.191:0/1000144118 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f194400f9e0 con 0x7f19480c69d0 2023-03-24T23:19:55.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.866+0000 7f193effd700 1 -- 172.21.15.191:0/1000144118 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f194400fb40 con 0x7f19480c69d0 2023-03-24T23:19:55.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.867+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f194815da30 con 0x7f19480c69d0 2023-03-24T23:19:55.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.867+0000 7f193effd700 1 -- 172.21.15.191:0/1000144118 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1944014440 con 0x7f19480c69d0 2023-03-24T23:19:55.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.867+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19480cc2e0 con 0x7f19480c69d0 2023-03-24T23:19:55.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.868+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1948155590 con 0x7f19480c69d0 2023-03-24T23:19:55.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.868+0000 7f193effd700 1 -- 172.21.15.191:0/1000144118 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f19440145a0 con 0x7f19480c69d0 2023-03-24T23:19:55.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.868+0000 7f193effd700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f192c069de0 0x7f192c06c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:55.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.868+0000 7f194e1d6700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f192c069de0 0x7f192c06c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:55.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.868+0000 7f193effd700 1 -- 172.21.15.191:0/1000144118 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f1944088470 con 0x7f19480c69d0 2023-03-24T23:19:55.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.869+0000 7f194e1d6700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f192c069de0 0x7f192c06c290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1938000fb0 tx=0x7f1938009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:55.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:55.873+0000 7f193effd700 1 -- 172.21.15.191:0/1000144118 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f1944054100 con 0x7f19480c69d0 2023-03-24T23:19:56.063 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.062+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f19480b0190 con 0x7f19480c69d0 2023-03-24T23:19:56.064 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.062+0000 7f193effd700 1 -- 172.21.15.191:0/1000144118 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v19) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1944057720 con 0x7f19480c69d0 2023-03-24T23:19:56.064 INFO:teuthology.orchestra.run.smithi191.stdout:pg_num: 1 2023-03-24T23:19:56.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f192c069de0 msgr2=0x7f192c06c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f192c069de0 0x7f192c06c290 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f1938000fb0 tx=0x7f1938009040 comp rx=0 tx=0).stop 2023-03-24T23:19:56.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19480c69d0 msgr2=0x7f19480c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19480c69d0 0x7f19480c4530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1944011aa0 tx=0x7f1944011e60 comp rx=0 tx=0).stop 2023-03-24T23:19:56.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 shutdown_connections 2023-03-24T23:19:56.066 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f192c069de0 0x7f192c06c290 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.067 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f19480c4a70 0x7f1948144da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.067 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1934004d90 0x7f194815f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.067 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 --2- 172.21.15.191:0/1000144118 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f19480c69d0 0x7f19480c4530 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.067 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.064+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 >> 172.21.15.191:0/1000144118 conn(0x7f19480d0b50 msgr2=0x7f19480b0470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:56.067 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.065+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 shutdown_connections 2023-03-24T23:19:56.067 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.065+0000 7f195043a700 1 -- 172.21.15.191:0/1000144118 wait complete. 2023-03-24T23:19:56.078 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2023-03-24T23:19:56.201 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.200+0000 7fe3b5606700 1 Processor -- start 2023-03-24T23:19:56.202 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.200+0000 7fe3b5606700 1 -- start start 2023-03-24T23:19:56.204 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3b5606700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe3b01584f0 0x7fe3b00b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.204 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3b5606700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b01588c0 con 0x7fe3b00b2fa0 2023-03-24T23:19:56.205 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3b5606700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b00b3420 con 0x7fe3b015bc50 2023-03-24T23:19:56.205 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3b5606700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b00ccb90 con 0x7fe3b01584f0 2023-03-24T23:19:56.205 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3af7fe700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe3b00b2fa0 0x7fe3b00cedc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57932/0 (socket says 172.21.15.191:57932) 2023-03-24T23:19:56.205 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3af7fe700 1 -- 172.21.15.191:0/3322395394 learned_addr learned my addr 172.21.15.191:0/3322395394 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:56.205 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4196777798 0 0) 0x7fe3b01588c0 con 0x7fe3b00b2fa0 2023-03-24T23:19:56.205 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3aeffd700 1 --2- 172.21.15.191:0/3322395394 >> v2:172.21.15.196:3300/0 conn(0x7fe3b01584f0 0x7fe3b00b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.205 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3980035b0 con 0x7fe3b00b2fa0 2023-03-24T23:19:56.206 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 <== mon.0 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069022063 0 0) 0x7fe3980035b0 con 0x7fe3b00b2fa0 2023-03-24T23:19:56.206 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 >> v2:172.21.15.196:3300/0 conn(0x7fe3b01584f0 msgr2=0x7fe3b00b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:56.206 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3adffb700 1 --2- 172.21.15.191:0/3322395394 >> v2:172.21.15.196:3300/0 conn(0x7fe3b01584f0 0x7fe3b00b2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.206 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 >> v1:172.21.15.196:6789/0 conn(0x7fe3b015bc50 legacy=0x7fe3b00d00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.206 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00cce70 con 0x7fe3b00b2fa0 2023-03-24T23:19:56.206 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.201+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 <== mon.0 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1537907258 0 0) 0x7fe3a80019f0 con 0x7fe3b00b2fa0 2023-03-24T23:19:56.206 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 >> v1:172.21.15.191:6789/0 conn(0x7fe3b00b2fa0 legacy=0x7fe3b00cedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.207 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe398003a00 0x7fe398003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.207 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3980043f0 0x7fe3980048b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.207 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3980050b0 0x7fe39800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.207 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00ccb90 con 0x7fe3980043f0 2023-03-24T23:19:56.207 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe39800c000 con 0x7fe398003a00 2023-03-24T23:19:56.207 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe39800c1c0 con 0x7fe3980050b0 2023-03-24T23:19:56.207 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3ae7fc700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3980043f0 0x7fe3980048b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.207 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3aeffd700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe398003a00 0x7fe398003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.208 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3af7fe700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3980050b0 0x7fe39800ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.208 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3ae7fc700 1 -- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3980050b0 msgr2=0x7fe39800ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:56.208 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3ae7fc700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3980050b0 0x7fe39800ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.208 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3ae7fc700 1 -- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe398003a00 msgr2=0x7fe398003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:56.208 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3ae7fc700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe398003a00 0x7fe398003dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.208 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3ae7fc700 1 -- 172.21.15.191:0/3322395394 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3980035b0 con 0x7fe3980043f0 2023-03-24T23:19:56.208 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3af7fe700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3980050b0 0x7fe39800ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-24T23:19:56.209 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3af7fe700 1 -- 172.21.15.191:0/3322395394 reap_dead start 2023-03-24T23:19:56.209 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3ae7fc700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3980043f0 0x7fe3980048b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00017c0 tx=0x7fe398005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9ea6bd0b2f2efe7 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.209 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a00052d0 con 0x7fe3980043f0 2023-03-24T23:19:56.209 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a0005430 con 0x7fe3980043f0 2023-03-24T23:19:56.209 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.202+0000 7fe3adffb700 1 -- 172.21.15.191:0/3322395394 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a00038c0 con 0x7fe3980043f0 2023-03-24T23:19:56.209 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.203+0000 7fe3b5606700 1 -- 172.21.15.191:0/3322395394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3980043f0 msgr2=0x7fe3980048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.209 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.203+0000 7fe3b5606700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3980043f0 0x7fe3980048b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe3a00017c0 tx=0x7fe398005880 comp rx=0 tx=0).stop 2023-03-24T23:19:56.209 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.203+0000 7fe3b5606700 1 -- 172.21.15.191:0/3322395394 shutdown_connections 2023-03-24T23:19:56.210 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.203+0000 7fe3b5606700 1 --2- 172.21.15.191:0/3322395394 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3980043f0 0x7fe3980048b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.210 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.203+0000 7fe3b5606700 1 -- 172.21.15.191:0/3322395394 >> 172.21.15.191:0/3322395394 conn(0x7fe3b00d0fe0 msgr2=0x7fe3b00d13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:56.210 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.203+0000 7fe3b5606700 1 -- 172.21.15.191:0/3322395394 shutdown_connections 2023-03-24T23:19:56.210 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.203+0000 7fe3b5606700 1 -- 172.21.15.191:0/3322395394 wait complete. 2023-03-24T23:19:56.212 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.204+0000 7fe3b5606700 1 Processor -- start 2023-03-24T23:19:56.212 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.204+0000 7fe3b5606700 1 -- start start 2023-03-24T23:19:56.212 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3b5606700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3980043f0 0x7fe3b006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.212 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3b5606700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3b01f6850 0x7fe3b01565c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.212 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3b5606700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b00b6710 0x7fe3b00c58a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3b5606700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b0155520 con 0x7fe3b00b6710 2023-03-24T23:19:56.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3b5606700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b0155800 con 0x7fe3980043f0 2023-03-24T23:19:56.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3b5606700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b0155ae0 con 0x7fe3b01f6850 2023-03-24T23:19:56.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b00b6710 0x7fe3b00c58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b00b6710 0x7fe3b00c58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50546/0 (socket says 172.21.15.191:50546) 2023-03-24T23:19:56.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 -- 172.21.15.191:0/3932055058 learned_addr learned my addr 172.21.15.191:0/3932055058 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:56.213 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3ae7fc700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3b01f6850 0x7fe3b01565c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3aeffd700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3980043f0 0x7fe3b006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 -- 172.21.15.191:0/3932055058 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3b01f6850 msgr2=0x7fe3b01565c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3b01f6850 0x7fe3b01565c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 -- 172.21.15.191:0/3932055058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3980043f0 msgr2=0x7fe3b006d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3980043f0 0x7fe3b006d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 -- 172.21.15.191:0/3932055058 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00cc7b0 con 0x7fe3b00b6710 2023-03-24T23:19:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.205+0000 7fe3af7fe700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b00b6710 0x7fe3b00c58a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8000f80 tx=0x7fe3a8008be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.214 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.206+0000 7fe3977fe700 1 -- 172.21.15.191:0/3932055058 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8006870 con 0x7fe3b00b6710 2023-03-24T23:19:56.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.206+0000 7fe3977fe700 1 -- 172.21.15.191:0/3932055058 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a80069d0 con 0x7fe3b00b6710 2023-03-24T23:19:56.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.206+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b00cd150 con 0x7fe3b00b6710 2023-03-24T23:19:56.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.206+0000 7fe3977fe700 1 -- 172.21.15.191:0/3932055058 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a8006d80 con 0x7fe3b00b6710 2023-03-24T23:19:56.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.206+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b00cc940 con 0x7fe3b00b6710 2023-03-24T23:19:56.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.206+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b006d6f0 con 0x7fe3b00b6710 2023-03-24T23:19:56.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.207+0000 7fe3977fe700 1 -- 172.21.15.191:0/3932055058 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7fe3a801e3b0 con 0x7fe3b00b6710 2023-03-24T23:19:56.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.207+0000 7fe3977fe700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe39c069de0 0x7fe39c06c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.215 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.207+0000 7fe3977fe700 1 -- 172.21.15.191:0/3932055058 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe3a8087b90 con 0x7fe3b00b6710 2023-03-24T23:19:56.216 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.207+0000 7fe3aeffd700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe39c069de0 0x7fe39c06c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.216 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.208+0000 7fe3aeffd700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe39c069de0 0x7fe39c06c290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009ce0 tx=0x7fe3a4009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.216 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.212+0000 7fe3977fe700 1 -- 172.21.15.191:0/3932055058 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7fe3a80537c0 con 0x7fe3b00b6710 2023-03-24T23:19:56.400 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.399+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe3b015aac0 con 0x7fe3b00b6710 2023-03-24T23:19:56.401 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.399+0000 7fe3977fe700 1 -- 172.21.15.191:0/3932055058 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v19) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe3a801a020 con 0x7fe3b00b6710 2023-03-24T23:19:56.401 INFO:teuthology.orchestra.run.smithi191.stdout:pg_num: 8 2023-03-24T23:19:56.403 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe39c069de0 msgr2=0x7fe39c06c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.403 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe39c069de0 0x7fe39c06c290 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fe3a4009ce0 tx=0x7fe3a4009210 comp rx=0 tx=0).stop 2023-03-24T23:19:56.403 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b00b6710 msgr2=0x7fe3b00c58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.403 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b00b6710 0x7fe3b00c58a0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8000f80 tx=0x7fe3a8008be0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.403 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 shutdown_connections 2023-03-24T23:19:56.404 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe39c069de0 0x7fe39c06c290 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.404 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe3b00b6710 0x7fe3b00c58a0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.404 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3980043f0 0x7fe3b006d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.404 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 --2- 172.21.15.191:0/3932055058 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3b01f6850 0x7fe3b01565c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.404 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.401+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 >> 172.21.15.191:0/3932055058 conn(0x7fe3b00d0fe0 msgr2=0x7fe3b015d090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:56.404 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.402+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 shutdown_connections 2023-03-24T23:19:56.404 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.402+0000 7fe3b5606700 1 -- 172.21.15.191:0/3932055058 wait complete. 2023-03-24T23:19:56.415 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-03-24T23:19:56.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.533+0000 7fe619165700 1 Processor -- start 2023-03-24T23:19:56.534 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.533+0000 7fe619165700 1 -- start start 2023-03-24T23:19:56.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe619165700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe6140ce810 0x7fe614163cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe619165700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6140cd370 con 0x7fe6140b10f0 2023-03-24T23:19:56.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe619165700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6140cf280 con 0x7fe61415d340 2023-03-24T23:19:56.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe619165700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe61415d850 con 0x7fe6140ce810 2023-03-24T23:19:56.536 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe612d9d700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7fe6140b10f0 0x7fe6140b14c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57950/0 (socket says 172.21.15.191:57950) 2023-03-24T23:19:56.537 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe61359e700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe6140ce810 0x7fe614163cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.537 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe612d9d700 1 -- 172.21.15.191:0/2024497740 learned_addr learned my addr 172.21.15.191:0/2024497740 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:56.537 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe61359e700 1 -- 172.21.15.191:0/2024497740 >> v1:172.21.15.196:6789/0 conn(0x7fe61415d340 legacy=0x7fe6140cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:56.537 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe61359e700 1 -- 172.21.15.191:0/2024497740 >> v1:172.21.15.191:6789/0 conn(0x7fe6140b10f0 legacy=0x7fe6140b14c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.537 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe61359e700 1 -- 172.21.15.191:0/2024497740 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6141591f0 con 0x7fe6140ce810 2023-03-24T23:19:56.537 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe61359e700 1 --2- 172.21.15.191:0/2024497740 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140ce810 0x7fe614163cc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009d90 tx=0x7fe60c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dde86cf81508d5d server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe611d9b700 1 -- 172.21.15.191:0/2024497740 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c00e070 con 0x7fe6140ce810 2023-03-24T23:19:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe611d9b700 1 -- 172.21.15.191:0/2024497740 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60c00bb90 con 0x7fe6140ce810 2023-03-24T23:19:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.534+0000 7fe611d9b700 1 -- 172.21.15.191:0/2024497740 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60c003a20 con 0x7fe6140ce810 2023-03-24T23:19:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.535+0000 7fe619165700 1 -- 172.21.15.191:0/2024497740 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140ce810 msgr2=0x7fe614163cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.535+0000 7fe619165700 1 --2- 172.21.15.191:0/2024497740 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140ce810 0x7fe614163cc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe60c009d90 tx=0x7fe60c00b330 comp rx=0 tx=0).stop 2023-03-24T23:19:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.535+0000 7fe619165700 1 -- 172.21.15.191:0/2024497740 shutdown_connections 2023-03-24T23:19:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.535+0000 7fe619165700 1 --2- 172.21.15.191:0/2024497740 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140ce810 0x7fe614163cc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.538 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.535+0000 7fe619165700 1 -- 172.21.15.191:0/2024497740 >> 172.21.15.191:0/2024497740 conn(0x7fe61406d880 msgr2=0x7fe6140cc0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:56.539 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.535+0000 7fe619165700 1 -- 172.21.15.191:0/2024497740 shutdown_connections 2023-03-24T23:19:56.539 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.535+0000 7fe619165700 1 -- 172.21.15.191:0/2024497740 wait complete. 2023-03-24T23:19:56.541 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.536+0000 7fe619165700 1 Processor -- start 2023-03-24T23:19:56.541 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.536+0000 7fe619165700 1 -- start start 2023-03-24T23:19:56.541 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe619165700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140b10f0 0x7fe6140c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.541 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe619165700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6140ce810 0x7fe6141526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.541 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe619165700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe61415d340 0x7fe6140b24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.541 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe619165700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6140c4160 con 0x7fe6140b10f0 2023-03-24T23:19:56.542 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe619165700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6140b2e80 con 0x7fe61415d340 2023-03-24T23:19:56.542 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe619165700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6140c3b10 con 0x7fe6140ce810 2023-03-24T23:19:56.542 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe612d9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140b10f0 0x7fe6140c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.542 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe612d9d700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140b10f0 0x7fe6140c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50556/0 (socket says 172.21.15.191:50556) 2023-03-24T23:19:56.542 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe612d9d700 1 -- 172.21.15.191:0/1202082720 learned_addr learned my addr 172.21.15.191:0/1202082720 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:56.542 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe61259c700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6140ce810 0x7fe6141526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.542 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe61359e700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe61415d340 0x7fe6140b24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.543 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe612d9d700 1 -- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6140ce810 msgr2=0x7fe6141526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.543 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe612d9d700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6140ce810 0x7fe6141526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.543 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe612d9d700 1 -- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe61415d340 msgr2=0x7fe6140b24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.543 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe612d9d700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe61415d340 0x7fe6140b24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.543 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe612d9d700 1 -- 172.21.15.191:0/1202082720 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c009a70 con 0x7fe6140b10f0 2023-03-24T23:19:56.543 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.537+0000 7fe61259c700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6140ce810 0x7fe6141526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:56.543 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.538+0000 7fe61359e700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe61415d340 0x7fe6140b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-03-24T23:19:56.543 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.538+0000 7fe612d9d700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140b10f0 0x7fe6140c6740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe60400c430 tx=0x7fe60400c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.544 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.538+0000 7fe5fb7fe700 1 -- 172.21.15.191:0/1202082720 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604005d70 con 0x7fe6140b10f0 2023-03-24T23:19:56.544 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.538+0000 7fe5fb7fe700 1 -- 172.21.15.191:0/1202082720 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6040070d0 con 0x7fe6140b10f0 2023-03-24T23:19:56.544 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.538+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6140ca0d0 con 0x7fe6140b10f0 2023-03-24T23:19:56.544 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.538+0000 7fe5fb7fe700 1 -- 172.21.15.191:0/1202082720 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe604007460 con 0x7fe6140b10f0 2023-03-24T23:19:56.544 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.538+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6140c9df0 con 0x7fe6140b10f0 2023-03-24T23:19:56.544 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.539+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6140ca3b0 con 0x7fe6140b10f0 2023-03-24T23:19:56.544 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.539+0000 7fe5fb7fe700 1 -- 172.21.15.191:0/1202082720 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7fe6040075c0 con 0x7fe6140b10f0 2023-03-24T23:19:56.544 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.540+0000 7fe5fb7fe700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe600069de0 0x7fe60006c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.545 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.540+0000 7fe61259c700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe600069de0 0x7fe60006c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.545 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.540+0000 7fe5fb7fe700 1 -- 172.21.15.191:0/1202082720 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7fe604087440 con 0x7fe6140b10f0 2023-03-24T23:19:56.545 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.540+0000 7fe61259c700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe600069de0 0x7fe60006c290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe61406bbb0 tx=0x7fe608009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.546 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.543+0000 7fe5fb7fe700 1 -- 172.21.15.191:0/1202082720 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7fe604053120 con 0x7fe6140b10f0 2023-03-24T23:19:56.734 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.733+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe6140c4440 con 0x7fe6140b10f0 2023-03-24T23:19:56.735 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.733+0000 7fe5fb7fe700 1 -- 172.21.15.191:0/1202082720 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v19) v1 ==== 84+0+126 (secure 0 0 0) 0x7fe604017090 con 0x7fe6140b10f0 2023-03-24T23:19:56.735 INFO:teuthology.orchestra.run.smithi191.stdout: 2023-03-24T23:19:56.735 INFO:teuthology.orchestra.run.smithi191.stdout:[ 2023-03-24T23:19:56.735 INFO:teuthology.orchestra.run.smithi191.stdout: { 2023-03-24T23:19:56.736 INFO:teuthology.orchestra.run.smithi191.stdout: "poolnum": 1, 2023-03-24T23:19:56.736 INFO:teuthology.orchestra.run.smithi191.stdout: "poolname": ".mgr" 2023-03-24T23:19:56.736 INFO:teuthology.orchestra.run.smithi191.stdout: }, 2023-03-24T23:19:56.736 INFO:teuthology.orchestra.run.smithi191.stdout: { 2023-03-24T23:19:56.736 INFO:teuthology.orchestra.run.smithi191.stdout: "poolnum": 2, 2023-03-24T23:19:56.736 INFO:teuthology.orchestra.run.smithi191.stdout: "poolname": "rbd" 2023-03-24T23:19:56.736 INFO:teuthology.orchestra.run.smithi191.stdout: } 2023-03-24T23:19:56.736 INFO:teuthology.orchestra.run.smithi191.stdout:] 2023-03-24T23:19:56.737 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe600069de0 msgr2=0x7fe60006c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe600069de0 0x7fe60006c290 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe61406bbb0 tx=0x7fe608009040 comp rx=0 tx=0).stop 2023-03-24T23:19:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140b10f0 msgr2=0x7fe6140c6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140b10f0 0x7fe6140c6740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe60400c430 tx=0x7fe60400c7f0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 shutdown_connections 2023-03-24T23:19:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe600069de0 0x7fe60006c290 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6140b10f0 0x7fe6140c6740 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe61415d340 0x7fe6140b24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.738 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 --2- 172.21.15.191:0/1202082720 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6140ce810 0x7fe6141526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.739 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.735+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 >> 172.21.15.191:0/1202082720 conn(0x7fe61406d880 msgr2=0x7fe6140b14c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:56.739 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.736+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 shutdown_connections 2023-03-24T23:19:56.739 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.736+0000 7fe619165700 1 -- 172.21.15.191:0/1202082720 wait complete. 2023-03-24T23:19:56.748 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-03-24T23:19:56.749 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-03-24T23:19:56.872 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.870+0000 7f8425b72700 1 Processor -- start 2023-03-24T23:19:56.872 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.870+0000 7f8425b72700 1 -- start start 2023-03-24T23:19:56.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f8425b72700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84200ce710 0x7f84200ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f8425b72700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84200b3490 0x7f84200b3910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f8425b72700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84200cdf40 con 0x7f8420070860 2023-03-24T23:19:56.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f8425b72700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84200cf0c0 con 0x7f84200b3490 2023-03-24T23:19:56.874 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f8425b72700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84201596b0 con 0x7f84200ce710 2023-03-24T23:19:56.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f841ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84200b3490 0x7f84200b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f841ffff700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7f84200b3490 0x7f84200b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50568/0 (socket says 172.21.15.191:50568) 2023-03-24T23:19:56.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f841ffff700 1 -- 172.21.15.191:0/2076589390 learned_addr learned my addr 172.21.15.191:0/2076589390 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:56.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.871+0000 7f841effd700 1 --2- 172.21.15.191:0/2076589390 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84200ce710 0x7f84200ceb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f841ffff700 1 -- 172.21.15.191:0/2076589390 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84200ce710 msgr2=0x7f84200ceb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:56.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f841ffff700 1 --2- 172.21.15.191:0/2076589390 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84200ce710 0x7f84200ceb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f841ffff700 1 -- 172.21.15.191:0/2076589390 >> v1:172.21.15.196:6789/0 conn(0x7f8420070860 legacy=0x7f8420070c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.875 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f841ffff700 1 -- 172.21.15.191:0/2076589390 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84200cd330 con 0x7f84200b3490 2023-03-24T23:19:56.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f841ffff700 1 --2- 172.21.15.191:0/2076589390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84200b3910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8418009e50 tx=0x7f841800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fed4183a31debbf server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f841e7fc700 1 -- 172.21.15.191:0/2076589390 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841800e070 con 0x7f84200b3490 2023-03-24T23:19:56.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f841e7fc700 1 -- 172.21.15.191:0/2076589390 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841800b910 con 0x7f84200b3490 2023-03-24T23:19:56.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f841e7fc700 1 -- 172.21.15.191:0/2076589390 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841800bcc0 con 0x7f84200b3490 2023-03-24T23:19:56.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f8425b72700 1 -- 172.21.15.191:0/2076589390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 msgr2=0x7f84200b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f8425b72700 1 --2- 172.21.15.191:0/2076589390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84200b3910 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8418009e50 tx=0x7f841800b330 comp rx=0 tx=0).stop 2023-03-24T23:19:56.876 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f8425b72700 1 -- 172.21.15.191:0/2076589390 shutdown_connections 2023-03-24T23:19:56.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f8425b72700 1 --2- 172.21.15.191:0/2076589390 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84200b3910 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f8425b72700 1 --2- 172.21.15.191:0/2076589390 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84200ce710 0x7f84200ceb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f8425b72700 1 -- 172.21.15.191:0/2076589390 >> 172.21.15.191:0/2076589390 conn(0x7f84200ccd40 msgr2=0x7f84200cf630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:56.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f8425b72700 1 -- 172.21.15.191:0/2076589390 shutdown_connections 2023-03-24T23:19:56.877 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.872+0000 7f8425b72700 1 -- 172.21.15.191:0/2076589390 wait complete. 2023-03-24T23:19:56.879 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.874+0000 7f8425b72700 1 Processor -- start 2023-03-24T23:19:56.879 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.874+0000 7f8425b72700 1 -- start start 2023-03-24T23:19:56.879 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.874+0000 7f8425b72700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8420070860 0x7f84200c6700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.879 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.874+0000 7f8425b72700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84201e6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.880 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.874+0000 7f8425b72700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84200ce710 0x7f8420148a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.880 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f8425b72700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84201e4b00 con 0x7f84200b3490 2023-03-24T23:19:56.880 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f8425b72700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84201e4dc0 con 0x7f84200ce710 2023-03-24T23:19:56.880 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f8425b72700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84201e50a0 con 0x7f8420070860 2023-03-24T23:19:56.880 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84201e6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.880 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84201e6aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50572/0 (socket says 172.21.15.191:50572) 2023-03-24T23:19:56.880 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 -- 172.21.15.191:0/3310741800 learned_addr learned my addr 172.21.15.191:0/3310741800 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:56.881 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841f7fe700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8420070860 0x7f84200c6700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.881 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 -- 172.21.15.191:0/3310741800 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8420070860 msgr2=0x7f84200c6700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.881 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8420070860 0x7f84200c6700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.881 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 -- 172.21.15.191:0/3310741800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84200ce710 msgr2=0x7f8420148a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:56.881 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84200ce710 0x7f8420148a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:56.881 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 -- 172.21.15.191:0/3310741800 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8418009b30 con 0x7f84200b3490 2023-03-24T23:19:56.881 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f841effd700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84201e6aa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f841000e870 tx=0x7f841000ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.881 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f8417fff700 1 -- 172.21.15.191:0/3310741800 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8410009590 con 0x7f84200b3490 2023-03-24T23:19:56.882 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f8417fff700 1 -- 172.21.15.191:0/3310741800 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f841000ee70 con 0x7f84200b3490 2023-03-24T23:19:56.882 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84200cd8f0 con 0x7f84200b3490 2023-03-24T23:19:56.882 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f8417fff700 1 -- 172.21.15.191:0/3310741800 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f841000ca50 con 0x7f84200b3490 2023-03-24T23:19:56.882 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.875+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84200cd610 con 0x7f84200b3490 2023-03-24T23:19:56.882 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.876+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84200cdbd0 con 0x7f84200b3490 2023-03-24T23:19:56.882 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.877+0000 7f8417fff700 1 -- 172.21.15.191:0/3310741800 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7f841000cbb0 con 0x7f84200b3490 2023-03-24T23:19:56.882 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.877+0000 7f8417fff700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f8400069de0 0x7f840006c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:56.882 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.877+0000 7f8417fff700 1 -- 172.21.15.191:0/3310741800 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5210+0+0 (secure 0 0 0) 0x7f8410087b80 con 0x7f84200b3490 2023-03-24T23:19:56.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.877+0000 7f841f7fe700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f8400069de0 0x7f840006c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:56.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.877+0000 7f841f7fe700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f8400069de0 0x7f840006c290 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8408000f60 tx=0x7f8408009040 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:56.883 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:56.881+0000 7f8417fff700 1 -- 172.21.15.191:0/3310741800 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7f8410057240 con 0x7f84200b3490 2023-03-24T23:19:57.071 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:57.069+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f8420074580 con 0x7f84200b3490 2023-03-24T23:19:58.022 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.020+0000 7f8417fff700 1 -- 172.21.15.191:0/3310741800 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v20) v1 ==== 138+0+0 (secure 0 0 0) 0x7f841000ce60 con 0x7f84200b3490 2023-03-24T23:19:58.026 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f8400069de0 msgr2=0x7f840006c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:58.026 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f8400069de0 0x7f840006c290 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f8408000f60 tx=0x7f8408009040 comp rx=0 tx=0).stop 2023-03-24T23:19:58.026 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 msgr2=0x7f84201e6aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:58.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84201e6aa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f841000e870 tx=0x7f841000ec30 comp rx=0 tx=0).stop 2023-03-24T23:19:58.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 shutdown_connections 2023-03-24T23:19:58.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7f8400069de0 0x7f840006c290 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f84200ce710 0x7f8420148a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8420070860 0x7f84200c6700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 --2- 172.21.15.191:0/3310741800 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f84200b3490 0x7f84201e6aa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 >> 172.21.15.191:0/3310741800 conn(0x7f84200ccd40 msgr2=0x7f842015b690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:58.027 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 shutdown_connections 2023-03-24T23:19:58.028 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.025+0000 7f8425b72700 1 -- 172.21.15.191:0/3310741800 wait complete. 2023-03-24T23:19:58.028 INFO:teuthology.orchestra.run.smithi191.stderr:pool 'cephfs_metadata' created 2023-03-24T23:19:58.043 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2023-03-24T23:19:58.176 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.175+0000 7fe677f0b700 1 Processor -- start 2023-03-24T23:19:58.176 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.175+0000 7fe677f0b700 1 -- start start 2023-03-24T23:19:58.178 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe677f0b700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe6700cf7e0 0x7fe6700ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:58.178 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe677f0b700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe6700cbd60 0x7fe670070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:58.178 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe677f0b700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe670160610 con 0x7fe6700cc1d0 2023-03-24T23:19:58.178 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe677f0b700 1 -- --> v2:172.21.15.191:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6701608f0 con 0x7fe6700cbd60 2023-03-24T23:19:58.178 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe677f0b700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe670160bd0 con 0x7fe6700cf7e0 2023-03-24T23:19:58.178 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe6754a6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe6700cbd60 0x7fe670070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:58.179 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe6754a6700 1 --2- >> v2:172.21.15.191:3300/0 conn(0x7fe6700cbd60 0x7fe670070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50580/0 (socket says 172.21.15.191:50580) 2023-03-24T23:19:58.179 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe6754a6700 1 -- 172.21.15.191:0/2896423762 learned_addr learned my addr 172.21.15.191:0/2896423762 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:58.179 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe675ca7700 1 --2- 172.21.15.191:0/2896423762 >> v2:172.21.15.196:3300/0 conn(0x7fe6700cf7e0 0x7fe6700ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:58.179 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe6754a6700 1 -- 172.21.15.191:0/2896423762 >> v2:172.21.15.196:3300/0 conn(0x7fe6700cf7e0 msgr2=0x7fe6700ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:58.179 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe6754a6700 1 --2- 172.21.15.191:0/2896423762 >> v2:172.21.15.196:3300/0 conn(0x7fe6700cf7e0 0x7fe6700ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.179 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe6754a6700 1 -- 172.21.15.191:0/2896423762 >> v1:172.21.15.196:6789/0 conn(0x7fe6700cc1d0 legacy=0x7fe67015d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:58.179 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe6754a6700 1 -- 172.21.15.191:0/2896423762 --> v2:172.21.15.191:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe670160eb0 con 0x7fe6700cbd60 2023-03-24T23:19:58.180 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe6754a6700 1 --2- 172.21.15.191:0/2896423762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cbd60 0x7fe670070fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe6700677a0 tx=0x7fe660009bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af1a05386e342ca2 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:58.180 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.176+0000 7fe674ca5700 1 -- 172.21.15.191:0/2896423762 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660011830 con 0x7fe6700cbd60 2023-03-24T23:19:58.180 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe674ca5700 1 -- 172.21.15.191:0/2896423762 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe660011990 con 0x7fe6700cbd60 2023-03-24T23:19:58.180 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe674ca5700 1 -- 172.21.15.191:0/2896423762 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660011d20 con 0x7fe6700cbd60 2023-03-24T23:19:58.182 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe677f0b700 1 -- 172.21.15.191:0/2896423762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cbd60 msgr2=0x7fe670070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:58.182 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe677f0b700 1 --2- 172.21.15.191:0/2896423762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cbd60 0x7fe670070fe0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe6700677a0 tx=0x7fe660009bd0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.182 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe677f0b700 1 -- 172.21.15.191:0/2896423762 shutdown_connections 2023-03-24T23:19:58.183 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe677f0b700 1 --2- 172.21.15.191:0/2896423762 >> v2:172.21.15.196:3300/0 conn(0x7fe6700cf7e0 0x7fe6700ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.183 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe677f0b700 1 --2- 172.21.15.191:0/2896423762 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cbd60 0x7fe670070fe0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.183 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe677f0b700 1 -- 172.21.15.191:0/2896423762 >> 172.21.15.191:0/2896423762 conn(0x7fe6700d0b50 msgr2=0x7fe6700cc940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:58.183 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe677f0b700 1 -- 172.21.15.191:0/2896423762 shutdown_connections 2023-03-24T23:19:58.183 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.177+0000 7fe677f0b700 1 -- 172.21.15.191:0/2896423762 wait complete. 2023-03-24T23:19:58.183 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.179+0000 7fe677f0b700 1 Processor -- start 2023-03-24T23:19:58.183 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.179+0000 7fe677f0b700 1 -- start start 2023-03-24T23:19:58.183 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe677f0b700 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6700cbd60 0x7fe6700c4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:58.184 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe677f0b700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cc1d0 0x7fe67015e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:58.184 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe677f0b700 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6700cf7e0 0x7fe6701f5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:58.184 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe677f0b700 1 -- --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe670151770 con 0x7fe6700cc1d0 2023-03-24T23:19:58.184 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe677f0b700 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6700c3960 con 0x7fe6700cf7e0 2023-03-24T23:19:58.184 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe677f0b700 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6701f6520 con 0x7fe6700cbd60 2023-03-24T23:19:58.184 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cc1d0 0x7fe67015e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:58.184 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 --2- >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cc1d0 0x7fe67015e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.191:3300/0 says I am v2:172.21.15.191:50586/0 (socket says 172.21.15.191:50586) 2023-03-24T23:19:58.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 -- 172.21.15.191:0/3361414087 learned_addr learned my addr 172.21.15.191:0/3361414087 (peer_addr_for_me v2:172.21.15.191:0/0) 2023-03-24T23:19:58.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe675ca7700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6700cbd60 0x7fe6700c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:58.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6764a8700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6700cf7e0 0x7fe6701f5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:58.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 -- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6700cbd60 msgr2=0x7fe6700c4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:58.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6700cbd60 0x7fe6700c4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 -- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6700cf7e0 msgr2=0x7fe6701f5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:58.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6700cf7e0 0x7fe6701f5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:58.185 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 -- 172.21.15.191:0/3361414087 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe660009550 con 0x7fe6700cc1d0 2023-03-24T23:19:58.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6764a8700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6700cf7e0 0x7fe6701f5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-03-24T23:19:58.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6754a6700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cc1d0 0x7fe67015e7e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe660000c00 tx=0x7fe660004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:58.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6667fc700 1 -- 172.21.15.191:0/3361414087 <== mon.0 v2:172.21.15.191:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe660004480 con 0x7fe6700cc1d0 2023-03-24T23:19:58.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6667fc700 1 -- 172.21.15.191:0/3361414087 <== mon.0 v2:172.21.15.191:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe66001f070 con 0x7fe6700cc1d0 2023-03-24T23:19:58.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe670161470 con 0x7fe6700cc1d0 2023-03-24T23:19:58.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe6667fc700 1 -- 172.21.15.191:0/3361414087 <== mon.0 v2:172.21.15.191:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6600038e0 con 0x7fe6700cc1d0 2023-03-24T23:19:58.186 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.180+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe670161190 con 0x7fe6700cc1d0 2023-03-24T23:19:58.187 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.181+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6700cfbb0 con 0x7fe6700cc1d0 2023-03-24T23:19:58.187 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.182+0000 7fe6667fc700 1 -- 172.21.15.191:0/3361414087 <== mon.0 v2:172.21.15.191:3300/0 4 ==== mgrmap(e 5) v1 ==== 87609+0+0 (secure 0 0 0) 0x7fe660021030 con 0x7fe6700cc1d0 2023-03-24T23:19:58.187 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.182+0000 7fe6667fc700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe654069de0 0x7fe65406c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:58.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.182+0000 7fe6667fc700 1 -- 172.21.15.191:0/3361414087 <== mon.0 v2:172.21.15.191:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5580+0+0 (secure 0 0 0) 0x7fe660089140 con 0x7fe6700cc1d0 2023-03-24T23:19:58.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.182+0000 7fe675ca7700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe654069de0 0x7fe65406c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:58.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.182+0000 7fe675ca7700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe654069de0 0x7fe65406c290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe67015fd20 tx=0x7fe66c009210 comp rx=0 tx=0).ready entity=mgr.4107 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-03-24T23:19:58.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.187+0000 7fe6667fc700 1 -- 172.21.15.191:0/3361414087 <== mon.0 v2:172.21.15.191:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176450 (secure 0 0 0) 0x7fe660054c60 con 0x7fe6700cc1d0 2023-03-24T23:19:58.287 DEBUG:teuthology.orchestra.run.smithi191:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-24T23:19:58.289 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-03-24T23:19:58.381 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:58.380+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7fe6700ceae0 con 0x7fe6700cc1d0 2023-03-24T23:19:59.029 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.027+0000 7fe6667fc700 1 -- 172.21.15.191:0/3361414087 <== mon.0 v2:172.21.15.191:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v21) v1 ==== 171+0+0 (secure 0 0 0) 0x7fe660058280 con 0x7fe6700cc1d0 2023-03-24T23:19:59.034 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.032+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe654069de0 msgr2=0x7fe65406c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:59.034 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.032+0000 7fe677f0b700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe654069de0 0x7fe65406c290 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe67015fd20 tx=0x7fe66c009210 comp rx=0 tx=0).stop 2023-03-24T23:19:59.034 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.032+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cc1d0 msgr2=0x7fe67015e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:59.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.032+0000 7fe677f0b700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cc1d0 0x7fe67015e7e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe660000c00 tx=0x7fe660004750 comp rx=0 tx=0).stop 2023-03-24T23:19:59.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.033+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 shutdown_connections 2023-03-24T23:19:59.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.033+0000 7fe677f0b700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:6832/97376,v1:172.21.15.191:6833/97376] conn(0x7fe654069de0 0x7fe65406c290 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:59.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.033+0000 7fe677f0b700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6700cf7e0 0x7fe6701f5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:59.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.033+0000 7fe677f0b700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6700cbd60 0x7fe6700c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:59.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.033+0000 7fe677f0b700 1 --2- 172.21.15.191:0/3361414087 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7fe6700cc1d0 0x7fe67015e7e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:59.035 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.033+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 >> 172.21.15.191:0/3361414087 conn(0x7fe6700d0b50 msgr2=0x7fe6700b1a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-03-24T23:19:59.036 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.033+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 shutdown_connections 2023-03-24T23:19:59.036 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.034+0000 7fe677f0b700 1 -- 172.21.15.191:0/3361414087 wait complete. 2023-03-24T23:19:59.042 INFO:teuthology.orchestra.run.smithi191.stderr:pool 'cephfs_data' created 2023-03-24T23:19:59.061 DEBUG:teuthology.orchestra.run.smithi191:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-03-24T23:19:59.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.187+0000 7f8ae50f1700 1 Processor -- start 2023-03-24T23:19:59.188 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.187+0000 7f8ae50f1700 1 -- start start 2023-03-24T23:19:59.191 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.187+0000 7f8ae50f1700 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8ae00716f0 0x7f8ae00ce4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:59.191 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8ae50f1700 1 -- --> v1:172.21.15.191:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae00cfd10 con 0x7f8ae015cfd0 2023-03-24T23:19:59.191 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8ae50f1700 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ae015d470 con 0x7f8ae00cda00 2023-03-24T23:19:59.191 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8ae50f1700 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ae015e080 con 0x7f8ae00716f0 2023-03-24T23:19:59.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8ade59c700 1 --1- >> v1:172.21.15.191:6789/0 conn(0x7f8ae015cfd0 0x7f8ae00cddd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.191:6789/0 says I am v1:172.21.15.191:57992/0 (socket says 172.21.15.191:57992) 2023-03-24T23:19:59.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8ade59c700 1 -- 172.21.15.191:0/2745379077 learned_addr learned my addr 172.21.15.191:0/2745379077 (peer_addr_for_me v1:172.21.15.191:0/0) 2023-03-24T23:19:59.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8adf59e700 1 --2- 172.21.15.191:0/2745379077 >> v2:172.21.15.196:3300/0 conn(0x7f8ae00716f0 0x7f8ae00ce4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-03-24T23:19:59.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 <== mon.0 v1:172.21.15.191:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077975956 0 0) 0x7f8ae00cfd10 con 0x7f8ae015cfd0 2023-03-24T23:19:59.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 --> v1:172.21.15.191:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ac80035b0 con 0x7f8ae015cfd0 2023-03-24T23:19:59.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 <== mon.0 v1:172.21.15.191:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 367499624 0 0) 0x7f8ac80035b0 con 0x7f8ae015cfd0 2023-03-24T23:19:59.192 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 >> v2:172.21.15.196:3300/0 conn(0x7f8ae00716f0 msgr2=0x7f8ae00ce4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-03-24T23:19:59.193 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 --2- 172.21.15.191:0/2745379077 >> v2:172.21.15.196:3300/0 conn(0x7f8ae00716f0 0x7f8ae00ce4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-03-24T23:19:59.193 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 >> v1:172.21.15.196:6789/0 conn(0x7f8ae00cda00 legacy=0x7f8ae0070fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:59.193 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 --> v1:172.21.15.191:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ae015d750 con 0x7f8ae015cfd0 2023-03-24T23:19:59.193 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 <== mon.0 v1:172.21.15.191:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 1537907258 0 0) 0x7f8ad4001670 con 0x7f8ae015cfd0 2023-03-24T23:19:59.193 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 >> v1:172.21.15.191:6789/0 conn(0x7f8ae015cfd0 legacy=0x7f8ae00cddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-03-24T23:19:59.193 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 --2- 172.21.15.191:0/2745379077 >> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] conn(0x7f8ac8003a00 0x7f8ac8003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:59.193 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 --2- 172.21.15.191:0/2745379077 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ac8004850 0x7f8ac8004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:59.194 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 --2- 172.21.15.191:0/2745379077 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ac8005200 0x7f8ac800ba10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-03-24T23:19:59.194 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 --> [v2:172.21.15.191:3300/0,v1:172.21.15.191:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ae015e080 con 0x7f8ac8003a00 2023-03-24T23:19:59.194 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac800c110 con 0x7f8ac8005200 2023-03-24T23:19:59.194 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8addd9b700 1 -- 172.21.15.191:0/2745379077 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac800c2d0 con 0x7f8ac8004850 2023-03-24T23:19:59.194 INFO:teuthology.orchestra.run.smithi191.stderr:2023-03-24T23:19:59.188+0000 7f8adf59e700 1 --2- 172.21.15.191:0/2745379077 >> v2:172.21.15.196:3300/0 conn(0x7f8ae00716f0 0x7f8ae00ce4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_do