2024-06-09T21:43:42.109 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-09T21:43:42.109 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind} --name teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204 --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/orig.config.yaml 2024-06-09T21:43:42.159 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-09T21:43:42.257 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204 branch: squid description: fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind} email: ceph-qa@ceph.com first_in_suite: false job_id: '7747204' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd session_timeout: 300 conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mds beacon grace: 60 mds heartbeat grace: 60 osd heartbeat grace: 60 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds valgrind exit: true mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 mon osd crush smoke test: false osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd fast shutdown: false osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow request - SLOW_OPS - MON_DOWN sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes watchdog: daemon_restart: normal ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: valgrind: - --tool=memcheck - --leak-check=full - --show-reachable=yes install: ceph: debuginfo: true flavor: default sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 178 sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba sleep_before_teardown: 0 subset: 130/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7b7a5d375cf4e4afab96b9d4d229b699ae8304bb syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCB91KijbRW8o/02HQs4ZPWIzY7nw/kn9ggUOIz4B+uLmnaVz4tJHt/SmsTFqAn1hQzzQys5R1jzk5qyn3zGbVM= smithi148.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMF/mae4dPj95xN4d/T7HfQkyRz/7xjOcRo9mLSElNtAnQmqDkXpqzhCHMBf1NSH+KDwdwBWx3o/bZ0/T744NgQ= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-09_21:24:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-09T21:43:42.257 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa; will attempt to use it 2024-06-09T21:43:42.258 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks 2024-06-09T21:43:42.259 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-09T21:43:42.261 INFO:teuthology.task.internal:Checking packages... 2024-06-09T21:43:42.287 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3d89accfe3201286f000e065a1faea2a88c7a6ba' 2024-06-09T21:43:42.287 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-09T21:43:42.287 INFO:teuthology.packaging:ref: None 2024-06-09T21:43:42.287 INFO:teuthology.packaging:tag: None 2024-06-09T21:43:42.288 INFO:teuthology.packaging:branch: squid 2024-06-09T21:43:42.288 INFO:teuthology.packaging:sha1: 3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T21:43:42.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-06-09T21:43:42.539 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2970.g7b7a5d37 2024-06-09T21:43:42.541 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-09T21:43:42.548 INFO:teuthology.task.internal:no buildpackages task found 2024-06-09T21:43:42.548 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-09T21:43:42.580 INFO:teuthology.task.internal:Saving configuration 2024-06-09T21:43:42.596 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-09T21:43:42.603 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-09T21:43:42.628 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-09 21:38:37.958790', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCB91KijbRW8o/02HQs4ZPWIzY7nw/kn9ggUOIz4B+uLmnaVz4tJHt/SmsTFqAn1hQzzQys5R1jzk5qyn3zGbVM='} 2024-06-09T21:43:42.648 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi148.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-09 21:38:37.960789', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMF/mae4dPj95xN4d/T7HfQkyRz/7xjOcRo9mLSElNtAnQmqDkXpqzhCHMBf1NSH+KDwdwBWx3o/bZ0/T744NgQ='} 2024-06-09T21:43:42.648 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-09T21:43:42.656 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-06-09T21:43:42.656 INFO:teuthology.task.internal:roles: ubuntu@smithi148.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-06-09T21:43:42.656 INFO:teuthology.run_tasks:Running task console_log... 2024-06-09T21:43:42.713 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f5f8db97be0>, signals=[15]) 2024-06-09T21:43:42.713 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-09T21:43:42.720 INFO:teuthology.task.internal:Opening connections... 2024-06-09T21:43:42.720 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T21:43:42.721 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:43:42.797 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T21:43:42.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:43:42.868 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-09T21:43:42.874 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-06-09T21:43:42.893 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-06-09T21:43:42.893 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-06-09T21:43:42.951 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-06-09T21:43:42.951 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-06-09T21:43:42.951 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-06-09T21:43:42.951 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-09T21:43:42.952 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-09T21:43:42.953 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-06-09T21:43:42.970 DEBUG:teuthology.orchestra.run.smithi148:> uname -m 2024-06-09T21:43:42.988 INFO:teuthology.orchestra.run.smithi148.stdout:x86_64 2024-06-09T21:43:42.988 DEBUG:teuthology.orchestra.run.smithi148:> cat /etc/os-release 2024-06-09T21:43:43.045 INFO:teuthology.orchestra.run.smithi148.stdout:NAME="CentOS Stream" 2024-06-09T21:43:43.045 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION="9" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:ID="centos" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:ID_LIKE="rhel fedora" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:VERSION_ID="9" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:PLATFORM_ID="platform:el9" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:ANSI_COLOR="0;31" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:LOGO="fedora-logo-icon" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:HOME_URL="https://centos.org/" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-09T21:43:43.046 INFO:teuthology.orchestra.run.smithi148.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-09T21:43:43.047 INFO:teuthology.lock.ops:Updating smithi148.front.sepia.ceph.com on lock server 2024-06-09T21:43:43.070 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-09T21:43:43.078 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-09T21:43:43.083 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-09T21:43:43.083 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-06-09T21:43:43.086 DEBUG:teuthology.orchestra.run.smithi148:> test '!' -e /home/ubuntu/cephtest 2024-06-09T21:43:43.103 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-09T21:43:43.111 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-09T21:43:43.111 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-06-09T21:43:43.143 DEBUG:teuthology.orchestra.run.smithi148:> test -z $(ls -A /var/lib/ceph) 2024-06-09T21:43:43.259 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-09T21:43:43.308 INFO:teuthology.run_tasks:Running task kernel... 2024-06-09T21:43:43.325 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-09T21:43:43.325 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-09T21:43:43.326 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-09T21:43:43.326 DEBUG:teuthology.orchestra.run.smithi148:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-09T21:43:43.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T21:43:43.344 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-06-09T21:43:43.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T21:43:43.345 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-06-09T21:43:43.401 INFO:teuthology.orchestra.run.smithi148.stdout:5.14.0-437.el9.x86_64 2024-06-09T21:43:43.401 INFO:teuthology.task.kernel:Running kernel on smithi148: 5.14.0-437.el9.x86_64 2024-06-09T21:43:43.401 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum install -y kernel 2024-06-09T21:43:43.403 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-437.el9.x86_64 2024-06-09T21:43:43.403 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.14.0-437.el9.x86_64 2024-06-09T21:43:43.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-06-09T21:43:45.634 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-06-09T21:43:45.652 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-06-09T21:43:47.703 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-06-09T21:43:50.135 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-06-09T21:43:51.904 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.2 MB 00:00 2024-06-09T21:43:53.562 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 17 kB 00:00 2024-06-09T21:43:54.590 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-06-09T21:43:56.402 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 2.3 MB/s | 6.2 MB 00:02 2024-06-09T21:43:57.843 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 17 kB 00:00 2024-06-09T21:43:59.087 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-06-09T21:43:59.996 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-06-09T21:44:02.077 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-09T21:44:02.078 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-09T21:44:02.174 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-06-09T21:44:02.181 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-06-09T21:44:02.182 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-06-09T21:44:02.183 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-06-09T21:44:02.183 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:44:02.183 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 80 M 2024-06-09T21:44:02.183 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 124 M 2024-06-09T21:44:02.183 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-06-09T21:44:02.537 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:44:02.538 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:44:02.541 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:44:02.706 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 3.5 MB/s | 893 kB 00:00 2024-06-09T21:44:03.090 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-06-09T21:44:03.423 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 40 MB/s | 29 MB 00:00 2024-06-09T21:44:03.807 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 26 MB/s | 35 MB 00:01 2024-06-09T21:44:03.809 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-06-09T21:44:03.810 INFO:teuthology.orchestra.run.smithi032.stdout:Total 49 MB/s | 80 MB 00:01 2024-06-09T21:44:04.276 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-06-09T21:44:04.539 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-06-09T21:44:04.950 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-06-09T21:44:04.950 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-06-09T21:44:05.920 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-06-09T21:44:05.920 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-06-09T21:44:06.266 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-09T21:44:06.266 INFO:teuthology.orchestra.run.smithi148.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-09T21:44:06.364 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-06-09T21:44:06.378 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout: Package Architecture Version Repository Size 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout:================================================================================ 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout:Install 4 Packages 2024-06-09T21:44:06.379 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:44:06.380 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 80 M 2024-06-09T21:44:06.380 INFO:teuthology.orchestra.run.smithi148.stdout:Installed size: 124 M 2024-06-09T21:44:06.380 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-06-09T21:44:06.608 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] kernel-core-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-core-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:44:06.612 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] kernel-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:44:06.613 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] kernel-modules-5.14.0-452.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/kernel-modules-5.14.0-452.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:44:06.877 INFO:teuthology.orchestra.run.smithi148.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 2.5 MB/s | 893 kB 00:00 2024-06-09T21:44:07.328 INFO:teuthology.orchestra.run.smithi148.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-06-09T21:44:07.903 INFO:teuthology.orchestra.run.smithi148.stdout:(3/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 25 MB/s | 35 MB 00:01 2024-06-09T21:44:08.303 INFO:teuthology.orchestra.run.smithi148.stdout:(4/4): kernel-modules-core-5.14.0-452.el9.x86_6 20 MB/s | 29 MB 00:01 2024-06-09T21:44:08.306 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-06-09T21:44:08.306 INFO:teuthology.orchestra.run.smithi148.stdout:Total 42 MB/s | 80 MB 00:01 2024-06-09T21:44:08.448 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-06-09T21:44:09.003 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-06-09T21:44:09.096 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-09T21:44:09.192 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:44:09.345 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-06-09T21:44:09.345 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-06-09T21:44:10.322 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-06-09T21:44:10.322 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-06-09T21:44:11.965 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:44:12.107 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:44:12.744 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-06-09T21:44:13.373 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-09T21:44:13.470 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:44:16.126 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:44:16.265 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:44:17.207 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:44:17.263 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:20.992 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:44:21.202 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:22.375 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:25.920 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:53.319 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:53.361 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:53.726 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:53.726 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-09T21:44:53.726 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:44:53.726 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:44:54.532 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:54.533 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:44:54.533 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-06-09T21:44:54.533 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-09T21:44:54.533 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-09T21:44:54.533 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-09T21:44:54.533 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-09T21:44:54.533 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:44:54.533 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-06-09T21:44:54.916 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-06-09T21:44:54.983 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-09T21:44:54.984 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-09T21:44:54.984 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-09T21:44:54.984 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-09T21:44:54.984 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-09T21:44:54.985 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-09T21:44:54.985 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-06-09T21:44:54.985 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-06-09T21:44:55.038 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-09T21:44:55.039 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-09T21:44:55.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-06-09T21:44:55.078 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-06-09T21:44:55.078 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-06-09T21:44:55.078 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-06-09T21:44:55.078 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 68.el9 2024-06-09T21:44:55.078 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-06-09T21:44:55.078 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-09T21:44:55.078 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-06-09T21:44:55.078 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8264254 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-09T21:44:55.079 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-09T21:44:55.080 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-06-09T21:44:55.080 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:44:55.080 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-06-09T21:44:55.080 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-09T21:44:55.081 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 5.14.0-452.el9.x86_64 2024-06-09T21:44:55.081 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-09T21:44:55.289 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:55.330 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:55.645 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-06-09T21:44:55.726 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:55.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-09T21:44:55.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-09T21:44:55.726 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:44:56.510 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-06-09T21:44:56.923 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-06-09T21:44:56.976 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-09T21:44:57.000 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-06-09T21:44:57.002 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-06-09T21:44:57.022 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.ts8TKktlfE 2024-06-09T21:44:57.023 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ts8TKktlfE 2024-06-09T21:44:57.063 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-09T21:44:57.063 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-09T21:44:57.063 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-09T21:44:57.063 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-09T21:44:57.063 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-09T21:44:57.064 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-09T21:44:57.064 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi148.front.sepia.ceph.com, path=None, version=distro) 2024-06-09T21:44:57.064 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q kernel | sort -rV | head -n 1 2024-06-09T21:44:57.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.ts8TKktlfE 2024-06-09T21:44:57.116 INFO:teuthology.orchestra.run.smithi148.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-09T21:44:57.116 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-09T21:44:57.116 DEBUG:teuthology.orchestra.run.smithi148:> sudo rpm -qi grub2-tools 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Name : grub2-tools 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Epoch : 1 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Version : 2.06 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Release : 68.el9 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Architecture: x86_64 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Group : Unspecified 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Size : 8264254 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:License : GPLv3+ 2024-06-09T21:44:57.157 INFO:teuthology.orchestra.run.smithi148.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:Packager : builder@centos.org 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:Vendor : CentOS 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:Summary : Support tools for GRUB. 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:Description : 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-09T21:44:57.158 INFO:teuthology.orchestra.run.smithi148.stdout:hardware devices. 2024-06-09T21:44:57.159 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:44:57.159 INFO:teuthology.orchestra.run.smithi148.stdout:This subpackage provides tools for support of all platforms. 2024-06-09T21:44:57.160 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-09T21:44:57.160 INFO:teuthology.task.kernel:Updating grub on smithi148 to boot 5.14.0-452.el9.x86_64 2024-06-09T21:44:57.160 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-09T21:44:57.236 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.ts8TKktlfE is 6KB 2024-06-09T21:44:57.286 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.ts8TKktlfE 2024-06-09T21:44:57.301 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-06-09T21:44:57.364 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-09T21:44:57.364 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-09T21:44:57.364 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-09T21:44:57.364 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-09T21:44:57.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-09T21:44:57.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-06-09T21:44:57.807 INFO:teuthology.orchestra.run.smithi148.stderr:Generating grub configuration file ... 2024-06-09T21:44:59.179 INFO:teuthology.orchestra.run.smithi148.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-09T21:44:59.213 INFO:teuthology.orchestra.run.smithi148.stderr:done 2024-06-09T21:44:59.215 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-06-09T21:44:59.230 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.c0MqkuLJcG 2024-06-09T21:44:59.230 DEBUG:teuthology.orchestra.run.smithi148:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.c0MqkuLJcG 2024-06-09T21:44:59.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.c0MqkuLJcG 2024-06-09T21:44:59.427 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.c0MqkuLJcG is 6KB 2024-06-09T21:44:59.475 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.c0MqkuLJcG 2024-06-09T21:44:59.489 DEBUG:teuthology.orchestra.run.smithi148:> sudo /bin/ls /boot/loader/entries || true 2024-06-09T21:44:59.550 INFO:teuthology.orchestra.run.smithi148.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-09T21:44:59.551 INFO:teuthology.orchestra.run.smithi148.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-09T21:44:59.551 INFO:teuthology.orchestra.run.smithi148.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-09T21:44:59.551 INFO:teuthology.orchestra.run.smithi148.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-09T21:44:59.552 DEBUG:teuthology.orchestra.run.smithi148:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-09T21:44:59.693 DEBUG:teuthology.orchestra.run.smithi148:> sudo shutdown -r now 2024-06-09T21:45:27.554 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-09T21:45:27.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-06-09T21:45:27.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:45:29.698 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-09T21:45:29.698 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-06-09T21:45:29.699 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:46:04.191 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-06-09T21:46:13.200 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-06-09T21:46:13.201 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:46:27.574 DEBUG:teuthology.orchestra.remote:timed out 2024-06-09T21:46:31.582 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.148 2024-06-09T21:46:36.581 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-06-09T21:46:36.582 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:46:36.816 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-06-09T21:46:37.223 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-06-09T21:46:37.224 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-452.el9.x86_64"... 2024-06-09T21:46:37.224 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-06-09T21:46:37.242 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-452.el9.x86_64 2024-06-09T21:46:37.242 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-09T21:46:37.242 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-09T21:46:37.242 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-09T21:46:38.243 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-09T21:46:38.243 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-09T21:46:38.320 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-06-09T21:46:38.359 DEBUG:teuthology.parallel:result is None 2024-06-09T21:46:43.583 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-06-09T21:46:43.584 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:46:46.779 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-06-09T21:46:47.236 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-06-09T21:46:47.236 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-452.el9.x86_64"... 2024-06-09T21:46:47.236 DEBUG:teuthology.orchestra.run.smithi148:> uname -r 2024-06-09T21:46:47.251 INFO:teuthology.orchestra.run.smithi148.stdout:5.14.0-452.el9.x86_64 2024-06-09T21:46:47.251 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-09T21:46:47.251 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-09T21:46:47.251 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-09T21:46:48.252 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-09T21:46:48.252 DEBUG:teuthology.orchestra.run.smithi148:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-09T21:46:48.336 INFO:teuthology.orchestra.run.smithi148.stdout:ttyS1 2024-06-09T21:46:48.371 DEBUG:teuthology.parallel:result is None 2024-06-09T21:46:48.371 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-09T21:46:48.379 INFO:teuthology.task.internal:Creating test directory... 2024-06-09T21:46:48.379 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-09T21:46:48.382 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-09T21:46:48.399 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-09T21:46:48.406 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-09T21:46:48.414 INFO:teuthology.task.internal:Creating archive directory... 2024-06-09T21:46:48.415 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-09T21:46:48.442 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-09T21:46:48.469 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-09T21:46:48.476 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-09T21:46:48.476 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-09T21:46:48.506 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-09T21:46:48.543 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T21:46:48.546 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T21:46:48.555 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T21:46:48.556 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-09T21:46:48.558 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-09T21:46:48.565 INFO:teuthology.task.internal:Configuring sudo... 2024-06-09T21:46:48.565 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-09T21:46:48.600 DEBUG:teuthology.orchestra.run.smithi148:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-09T21:46:48.627 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-09T21:46:48.635 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-09T21:46:48.635 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-09T21:46:48.670 DEBUG:teuthology.orchestra.run.smithi148:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-09T21:46:48.686 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T21:46:48.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T21:46:48.821 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T21:46:48.909 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T21:46:48.953 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:46:48.953 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-09T21:46:49.021 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T21:46:49.059 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-09T21:46:49.112 DEBUG:teuthology.orchestra.run.smithi148:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T21:46:49.192 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-09T21:46:49.237 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:46:49.238 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-09T21:46:49.304 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-06-09T21:46:49.307 DEBUG:teuthology.orchestra.run.smithi148:> sudo service rsyslog restart 2024-06-09T21:46:49.341 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T21:46:49.383 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T21:46:49.808 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-09T21:46:49.816 INFO:teuthology.task.internal:Starting timer... 2024-06-09T21:46:49.816 INFO:teuthology.run_tasks:Running task pcp... 2024-06-09T21:46:49.826 INFO:teuthology.run_tasks:Running task selinux... 2024-06-09T21:46:49.834 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-06-09T21:46:49.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-06-09T21:46:49.898 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-06-09T21:46:49.900 DEBUG:teuthology.orchestra.run.smithi148:> sudo service auditd rotate 2024-06-09T21:46:49.964 INFO:teuthology.orchestra.run.smithi148.stdout:Rotating logs: 2024-06-09T21:46:49.965 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-09T21:46:49.966 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-06-09T21:46:50.014 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-06-09T21:46:50.014 DEBUG:teuthology.orchestra.run.smithi148:> /usr/sbin/getenforce 2024-06-09T21:46:50.060 INFO:teuthology.orchestra.run.smithi148.stdout:Permissive 2024-06-09T21:46:50.060 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi148.front.sepia.ceph.com': 'permissive'} 2024-06-09T21:46:50.061 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T21:46:50.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T21:46:50.090 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T21:46:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T21:46:50.125 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-09T21:46:50.126 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-06-09T21:46:50.158 DEBUG:teuthology.orchestra.run.smithi148:> sudo /usr/sbin/setenforce permissive 2024-06-09T21:46:50.191 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-09T21:46:50.200 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-09T21:46:50.203 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-09T21:46:50.345 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-09T21:46:50.356 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-09T21:46:50.357 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi148.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-09T21:51:19.146 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi148.front.sepia.ceph.com')] 2024-06-09T21:51:19.147 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-06-09T21:51:19.149 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:51:19.230 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-06-09T21:51:19.316 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-06-09T21:51:19.316 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-06-09T21:51:19.317 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi148.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-09T21:51:19.391 DEBUG:teuthology.orchestra.run.smithi148:> true 2024-06-09T21:51:19.473 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi148.front.sepia.ceph.com' 2024-06-09T21:51:19.473 INFO:teuthology.run_tasks:Running task clock... 2024-06-09T21:51:19.486 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-09T21:51:19.486 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-09T21:51:19.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T21:51:19.489 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-09T21:51:19.489 DEBUG:teuthology.orchestra.run.smithi148:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T21:51:19.522 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-09T21:51:19.538 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-09T21:51:19.542 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-09T21:51:19.557 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-09T21:51:19.566 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-06-09T21:51:19.578 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-06-09T21:51:19.581 INFO:teuthology.orchestra.run.smithi148.stderr:sudo: ntpd: command not found 2024-06-09T21:51:19.592 INFO:teuthology.orchestra.run.smithi148.stdout:506 Cannot talk to daemon 2024-06-09T21:51:19.593 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-09T21:51:19.605 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-09T21:51:19.608 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-09T21:51:19.619 INFO:teuthology.orchestra.run.smithi148.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-09T21:51:19.671 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-06-09T21:51:19.674 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T21:51:19.674 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-06-09T21:51:19.675 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T21:51:19.675 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T21:51:19.675 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T21:51:19.675 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T21:51:19.679 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-06-09T21:51:19.682 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T21:51:19.682 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-06-09T21:51:19.682 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T21:51:19.683 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T21:51:19.683 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T21:51:19.683 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T21:51:19.684 INFO:teuthology.run_tasks:Running task install... 2024-06-09T21:51:19.692 DEBUG:teuthology.task.install:project ceph 2024-06-09T21:51:19.692 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}} 2024-06-09T21:51:19.692 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'} 2024-06-09T21:51:19.692 INFO:teuthology.task.install:Using flavor: default 2024-06-09T21:51:19.698 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-06-09T21:51:19.698 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-06-09T21:51:19.698 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-09T21:51:19.699 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T21:51:19.701 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'tag': None, 'wait_for_package': False} 2024-06-09T21:51:19.701 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T21:51:19.849 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-09T21:51:19.849 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-09T21:51:19.913 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/ 2024-06-09T21:51:19.913 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2946.g3d89accf 2024-06-09T21:51:19.961 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-09T21:51:19.962 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:51:19.962 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-09T21:51:19.991 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-09T21:51:19.991 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-09T21:51:20.058 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-09T21:51:20.097 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/3d89accfe3201286f000e065a1faea2a88c7a6ba/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-09T21:51:20.097 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:51:20.097 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-09T21:51:20.123 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-06-09T21:51:20.123 DEBUG:teuthology.orchestra.run.smithi148:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3d89accfe3201286f000e065a1faea2a88c7a6ba/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-09T21:51:20.136 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-09T21:51:20.190 DEBUG:teuthology.orchestra.run.smithi148:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-09T21:51:20.201 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-06-09T21:51:20.203 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-06-09T21:51:20.264 DEBUG:teuthology.orchestra.run.smithi148:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-09T21:51:20.329 INFO:teuthology.orchestra.run.smithi148.stdout:check_obsoletes = 1 2024-06-09T21:51:20.330 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum clean all 2024-06-09T21:51:20.439 INFO:teuthology.orchestra.run.smithi032.stdout:45 files removed 2024-06-09T21:51:20.467 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-09T21:51:20.568 INFO:teuthology.orchestra.run.smithi148.stdout:45 files removed 2024-06-09T21:51:20.597 DEBUG:teuthology.orchestra.run.smithi148:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd ceph-base-debuginfo ceph-common-debuginfo ceph-immutable-object-cache-debuginfo ceph-radosgw-debuginfo ceph-test-debuginfo ceph-base-debuginfo ceph-mgr-debuginfo ceph-mds-debuginfo ceph-mon-debuginfo ceph-osd-debuginfo ceph-fuse-debuginfo librados-devel-debuginfo libcephfs2-debuginfo librados2-debuginfo librbd1-debuginfo python3-cephfs-debuginfo python3-rados-debuginfo python3-rbd-debuginfo python3-rgw-debuginfo rbd-fuse-debuginfo rbd-mirror-debuginfo rbd-nbd-debuginfo python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-06-09T21:51:21.173 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 201 kB/s | 83 kB 00:00 2024-06-09T21:51:21.292 INFO:teuthology.orchestra.run.smithi148.stdout:ceph packages for x86_64 198 kB/s | 83 kB 00:00 2024-06-09T21:51:21.549 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 36 kB/s | 11 kB 00:00 2024-06-09T21:51:21.675 INFO:teuthology.orchestra.run.smithi148.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-06-09T21:51:21.941 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-06-09T21:51:22.001 INFO:teuthology.orchestra.run.smithi148.stdout:ceph source packages 6.7 kB/s | 1.9 kB 00:00 2024-06-09T21:51:22.620 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-06-09T21:51:23.060 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 7.5 MB/s | 8.1 MB 00:01 2024-06-09T21:51:24.953 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-06-09T21:51:28.330 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - AppStream 4.2 MB/s | 19 MB 00:04 2024-06-09T21:51:29.319 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 9.9 MB/s | 6.2 MB 00:00 2024-06-09T21:51:31.146 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 17 kB 00:00 2024-06-09T21:51:32.174 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-06-09T21:51:33.591 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.2 MB 00:01 2024-06-09T21:51:35.388 INFO:teuthology.orchestra.run.smithi148.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 17 kB 00:00 2024-06-09T21:51:36.316 INFO:teuthology.orchestra.run.smithi148.stdout:Extra Packages for Enterprise Linux 27 MB/s | 21 MB 00:00 2024-06-09T21:51:37.655 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 47 kB/s | 1.7 kB 00:00 2024-06-09T21:51:40.074 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T21:51:40.075 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T21:51:40.104 INFO:teuthology.orchestra.run.smithi032.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-09T21:51:40.105 INFO:teuthology.orchestra.run.smithi032.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-09T21:51:40.105 INFO:teuthology.orchestra.run.smithi032.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-09T21:51:40.106 INFO:teuthology.orchestra.run.smithi032.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-09T21:51:40.108 INFO:teuthology.orchestra.run.smithi032.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-09T21:51:40.111 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-09T21:51:40.112 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-09T21:51:40.182 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-06-09T21:51:40.211 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================================== 2024-06-09T21:51:40.211 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-06-09T21:51:40.211 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================================== 2024-06-09T21:51:40.211 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-06-09T21:51:40.211 INFO:teuthology.orchestra.run.smithi032.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-09T21:51:40.211 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-09T21:51:40.211 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 69 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 451 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.0 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 53 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 20 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-09T21:51:40.212 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 M 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 288 M 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 187 M 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 692 M 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 221 k 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 27 k 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 18 k 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-09T21:51:40.213 INFO:teuthology.orchestra.run.smithi032.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 109 k 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 M 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 52 M 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-09T21:51:40.214 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 615 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 771 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.1 M 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 337 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 516 k 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T21:51:40.215 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 52 M 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.3 M 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-09T21:51:40.216 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 23 M 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-09T21:51:40.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-09T21:51:40.218 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-09T21:51:40.219 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-09T21:51:40.220 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-09T21:51:40.221 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-09T21:51:40.222 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-09T21:51:40.223 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-09T21:51:40.224 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================================== 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout:Install 170 Packages 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 11 Packages 2024-06-09T21:51:40.225 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:51:40.226 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 2.2 G 2024-06-09T21:51:40.226 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-06-09T21:51:41.240 INFO:teuthology.orchestra.run.smithi032.stdout:(1/181): ceph-19.0.0-2946.g3d89accf.el9.x86_64. 17 kB/s | 6.6 kB 00:00 2024-06-09T21:51:41.449 INFO:teuthology.orchestra.run.smithi032.stdout:(2/181): ceph-base-19.0.0-2946.g3d89accf.el9.x8 9.1 MB/s | 5.4 MB 00:00 2024-06-09T21:51:41.547 INFO:teuthology.orchestra.run.smithi148.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-06-09T21:51:43.865 INFO:teuthology.orchestra.run.smithi148.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T21:51:43.866 INFO:teuthology.orchestra.run.smithi148.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-06-09T21:51:43.891 INFO:teuthology.orchestra.run.smithi148.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-06-09T21:51:43.892 INFO:teuthology.orchestra.run.smithi148.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-06-09T21:51:43.893 INFO:teuthology.orchestra.run.smithi148.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-06-09T21:51:43.894 INFO:teuthology.orchestra.run.smithi148.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-06-09T21:51:43.895 INFO:teuthology.orchestra.run.smithi148.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-06-09T21:51:43.898 INFO:teuthology.orchestra.run.smithi148.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-06-09T21:51:43.899 INFO:teuthology.orchestra.run.smithi148.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-06-09T21:51:43.964 INFO:teuthology.orchestra.run.smithi148.stdout:Dependencies resolved. 2024-06-09T21:51:43.986 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================================== 2024-06-09T21:51:43.986 INFO:teuthology.orchestra.run.smithi148.stdout: Package Arch Version Repository Size 2024-06-09T21:51:43.986 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================================== 2024-06-09T21:51:43.986 INFO:teuthology.orchestra.run.smithi148.stdout:Installing: 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.6 k 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 69 M 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 451 M 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 890 k 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 6.0 M 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 145 k 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 53 M 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.5 M 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 142 k 2024-06-09T21:51:43.987 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 3.5 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 20 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 7.4 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 49 k 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 288 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 10 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 187 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 48 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 692 M 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 264 k 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 763 k 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 221 k 2024-06-09T21:51:43.988 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-shell x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 27 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-top noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 18 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libelf-devel x86_64 0.191-4.el9 appstream 22 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 32 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 747 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.4 M 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 127 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 109 k 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 M 2024-06-09T21:51:43.989 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 52 M 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 162 k 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 615 k 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 321 k 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 771 k 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 300 k 2024-06-09T21:51:43.990 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.1 M 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 99 k 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 337 k 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 85 k 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 516 k 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 52 M 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 171 k 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 1.3 M 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrading: 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager x86_64 1:1.47.91-1.el9 baseos 2.3 M 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-libnm x86_64 1:1.47.91-1.el9 baseos 1.8 M 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-team x86_64 1:1.47.91-1.el9 baseos 41 k 2024-06-09T21:51:43.991 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-tui x86_64 1:1.47.91-1.el9 baseos 248 k 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-debuginfod-client x86_64 0.191-4.el9 baseos 37 k 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libelf x86_64 0.191-4.el9 baseos 207 k 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libs x86_64 0.191-4.el9 baseos 260 k 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: librados2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.4 M 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 3.1 M 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout:Installing dependencies: 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 21 M 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-debuginfo x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 23 M 2024-06-09T21:51:43.992 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 32 k 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 2.4 M 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 249 k 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 4.7 M 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 17 M 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2946.g3d89accf.el9 ceph-noarch 16 k 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 25 k 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-09T21:51:43.993 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 163 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 501 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2 x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 5.3 M 2024-06-09T21:51:43.994 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-06-09T21:51:43.995 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 45 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common x86_64 2:19.0.0-2946.g3d89accf.el9 ceph 135 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-colorama noarch 0.4.6-3.el9 appstream 61 k 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-09T21:51:43.996 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-09T21:51:43.997 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-09T21:51:43.998 INFO:teuthology.orchestra.run.smithi148.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-09T21:51:43.999 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-09T21:51:44.000 INFO:teuthology.orchestra.run.smithi148.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout:Installing weak dependencies: 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction Summary 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================================== 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout:Install 170 Packages 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout:Upgrade 11 Packages 2024-06-09T21:51:44.001 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:51:44.002 INFO:teuthology.orchestra.run.smithi148.stdout:Total download size: 2.2 G 2024-06-09T21:51:44.002 INFO:teuthology.orchestra.run.smithi148.stdout:Downloading Packages: 2024-06-09T21:51:45.356 INFO:teuthology.orchestra.run.smithi148.stdout:(1/181): ceph-19.0.0-2946.g3d89accf.el9.x86_64. 9.3 kB/s | 6.6 kB 00:00 2024-06-09T21:51:46.990 INFO:teuthology.orchestra.run.smithi148.stdout:(2/181): ceph-base-19.0.0-2946.g3d89accf.el9.x8 2.3 MB/s | 5.4 MB 00:02 2024-06-09T21:51:48.106 INFO:teuthology.orchestra.run.smithi148.stdout:(3/181): ceph-common-19.0.0-2946.g3d89accf.el9. 7.6 MB/s | 21 MB 00:02 2024-06-09T21:51:49.390 INFO:teuthology.orchestra.run.smithi032.stdout:(3/181): ceph-common-19.0.0-2946.g3d89accf.el9. 2.6 MB/s | 21 MB 00:08 2024-06-09T21:51:50.573 INFO:teuthology.orchestra.run.smithi148.stdout:(4/181): ceph-debuginfo-19.0.0-2946.g3d89accf.e 9.5 MB/s | 23 MB 00:02 2024-06-09T21:51:50.740 INFO:teuthology.orchestra.run.smithi148.stdout:(5/181): ceph-fuse-19.0.0-2946.g3d89accf.el9.x8 5.2 MB/s | 890 kB 00:00 2024-06-09T21:51:51.382 INFO:teuthology.orchestra.run.smithi148.stdout:(6/181): ceph-fuse-debuginfo-19.0.0-2946.g3d89a 9.4 MB/s | 6.0 MB 00:00 2024-06-09T21:51:51.457 INFO:teuthology.orchestra.run.smithi148.stdout:(7/181): ceph-immutable-object-cache-19.0.0-294 1.9 MB/s | 145 kB 00:00 2024-06-09T21:51:51.732 INFO:teuthology.orchestra.run.smithi148.stdout:(8/181): ceph-immutable-object-cache-debuginfo- 5.3 MB/s | 1.5 MB 00:00 2024-06-09T21:51:51.999 INFO:teuthology.orchestra.run.smithi148.stdout:(9/181): ceph-mds-19.0.0-2946.g3d89accf.el9.x86 8.9 MB/s | 2.4 MB 00:00 2024-06-09T21:51:56.983 INFO:teuthology.orchestra.run.smithi148.stdout:(10/181): ceph-mds-debuginfo-19.0.0-2946.g3d89a 11 MB/s | 53 MB 00:04 2024-06-09T21:51:57.224 INFO:teuthology.orchestra.run.smithi148.stdout:(11/181): ceph-mgr-19.0.0-2946.g3d89accf.el9.x8 6.2 MB/s | 1.5 MB 00:00 2024-06-09T21:51:57.465 INFO:teuthology.orchestra.run.smithi032.stdout:(4/181): ceph-debuginfo-19.0.0-2946.g3d89accf.e 2.9 MB/s | 23 MB 00:08 2024-06-09T21:51:57.923 INFO:teuthology.orchestra.run.smithi032.stdout:(5/181): ceph-fuse-19.0.0-2946.g3d89accf.el9.x8 1.9 MB/s | 890 kB 00:00 2024-06-09T21:51:59.258 INFO:teuthology.orchestra.run.smithi148.stdout:(12/181): ceph-mgr-debuginfo-19.0.0-2946.g3d89a 9.9 MB/s | 20 MB 00:02 2024-06-09T21:51:59.783 INFO:teuthology.orchestra.run.smithi148.stdout:(13/181): ceph-mon-19.0.0-2946.g3d89accf.el9.x8 8.9 MB/s | 4.7 MB 00:00 2024-06-09T21:52:00.340 INFO:teuthology.orchestra.run.smithi032.stdout:(6/181): ceph-base-debuginfo-19.0.0-2946.g3d89a 3.5 MB/s | 69 MB 00:19 2024-06-09T21:52:00.423 INFO:teuthology.orchestra.run.smithi032.stdout:(7/181): ceph-fuse-debuginfo-19.0.0-2946.g3d89a 2.4 MB/s | 6.0 MB 00:02 2024-06-09T21:52:00.524 INFO:teuthology.orchestra.run.smithi032.stdout:(8/181): ceph-immutable-object-cache-19.0.0-294 790 kB/s | 145 kB 00:00 2024-06-09T21:52:01.074 INFO:teuthology.orchestra.run.smithi032.stdout:(9/181): ceph-immutable-object-cache-debuginfo- 2.2 MB/s | 1.5 MB 00:00 2024-06-09T21:52:01.558 INFO:teuthology.orchestra.run.smithi032.stdout:(10/181): ceph-mds-19.0.0-2946.g3d89accf.el9.x8 2.3 MB/s | 2.4 MB 00:01 2024-06-09T21:52:02.241 INFO:teuthology.orchestra.run.smithi032.stdout:(11/181): ceph-mgr-19.0.0-2946.g3d89accf.el9.x8 2.2 MB/s | 1.5 MB 00:00 2024-06-09T21:52:07.825 INFO:teuthology.orchestra.run.smithi032.stdout:(12/181): ceph-mgr-debuginfo-19.0.0-2946.g3d89a 3.6 MB/s | 20 MB 00:05 2024-06-09T21:52:08.691 INFO:teuthology.orchestra.run.smithi032.stdout:(13/181): ceph-mon-19.0.0-2946.g3d89accf.el9.x8 5.4 MB/s | 4.7 MB 00:00 2024-06-09T21:52:10.207 INFO:teuthology.orchestra.run.smithi148.stdout:(14/181): ceph-mon-debuginfo-19.0.0-2946.g3d89a 9.5 MB/s | 99 MB 00:10 2024-06-09T21:52:12.882 INFO:teuthology.orchestra.run.smithi148.stdout:(15/181): ceph-base-debuginfo-19.0.0-2946.g3d89 2.4 MB/s | 69 MB 00:28 2024-06-09T21:52:13.299 INFO:teuthology.orchestra.run.smithi148.stdout:(16/181): ceph-osd-19.0.0-2946.g3d89accf.el9.x8 5.4 MB/s | 17 MB 00:03 2024-06-09T21:52:15.183 INFO:teuthology.orchestra.run.smithi148.stdout:(17/181): ceph-radosgw-19.0.0-2946.g3d89accf.el 5.4 MB/s | 10 MB 00:01 2024-06-09T21:52:18.875 INFO:teuthology.orchestra.run.smithi032.stdout:(14/181): ceph-mon-debuginfo-19.0.0-2946.g3d89a 9.7 MB/s | 99 MB 00:10 2024-06-09T21:52:19.342 INFO:teuthology.orchestra.run.smithi032.stdout:(15/181): ceph-mds-debuginfo-19.0.0-2946.g3d89a 2.9 MB/s | 53 MB 00:18 2024-06-09T21:52:20.967 INFO:teuthology.orchestra.run.smithi032.stdout:(16/181): ceph-osd-19.0.0-2946.g3d89accf.el9.x8 8.0 MB/s | 17 MB 00:02 2024-06-09T21:52:22.209 INFO:teuthology.orchestra.run.smithi032.stdout:(17/181): ceph-radosgw-19.0.0-2946.g3d89accf.el 8.2 MB/s | 10 MB 00:01 2024-06-09T21:52:50.682 INFO:teuthology.orchestra.run.smithi032.stdout:(18/181): ceph-radosgw-debuginfo-19.0.0-2946.g3 6.6 MB/s | 187 MB 00:28 2024-06-09T21:52:50.782 INFO:teuthology.orchestra.run.smithi032.stdout:(19/181): ceph-selinux-19.0.0-2946.g3d89accf.el 252 kB/s | 25 kB 00:00 2024-06-09T21:52:52.782 INFO:teuthology.orchestra.run.smithi148.stdout:(18/181): ceph-common-debuginfo-19.0.0-2946.g3d 6.9 MB/s | 451 MB 01:05 2024-06-09T21:52:52.924 INFO:teuthology.orchestra.run.smithi148.stdout:(19/181): ceph-selinux-19.0.0-2946.g3d89accf.el 178 kB/s | 25 kB 00:00 2024-06-09T21:52:56.823 INFO:teuthology.orchestra.run.smithi032.stdout:(20/181): ceph-test-19.0.0-2946.g3d89accf.el9.x 8.0 MB/s | 48 MB 00:06 2024-06-09T21:53:00.890 INFO:teuthology.orchestra.run.smithi148.stdout:(20/181): ceph-test-19.0.0-2946.g3d89accf.el9.x 6.1 MB/s | 48 MB 00:07 2024-06-09T21:53:06.548 INFO:teuthology.orchestra.run.smithi032.stdout:(21/181): ceph-common-debuginfo-19.0.0-2946.g3d 5.3 MB/s | 451 MB 01:25 2024-06-09T21:53:06.881 INFO:teuthology.orchestra.run.smithi032.stdout:(22/181): cephfs-mirror-19.0.0-2946.g3d89accf.e 665 kB/s | 221 kB 00:00 2024-06-09T21:53:07.306 INFO:teuthology.orchestra.run.smithi032.stdout:(23/181): cephfs-shell-19.0.0-2946.g3d89accf.el 64 kB/s | 27 kB 00:00 2024-06-09T21:53:07.374 INFO:teuthology.orchestra.run.smithi032.stdout:(24/181): libcephfs-devel-19.0.0-2946.g3d89accf 477 kB/s | 32 kB 00:00 2024-06-09T21:53:07.574 INFO:teuthology.orchestra.run.smithi032.stdout:(25/181): libcephfs2-19.0.0-2946.g3d89accf.el9. 3.7 MB/s | 747 kB 00:00 2024-06-09T21:53:07.981 INFO:teuthology.orchestra.run.smithi148.stdout:(21/181): ceph-radosgw-debuginfo-19.0.0-2946.g3 3.5 MB/s | 187 MB 00:52 2024-06-09T21:53:08.239 INFO:teuthology.orchestra.run.smithi148.stdout:(22/181): cephfs-mirror-19.0.0-2946.g3d89accf.e 859 kB/s | 221 kB 00:00 2024-06-09T21:53:08.348 INFO:teuthology.orchestra.run.smithi148.stdout:(23/181): cephfs-shell-19.0.0-2946.g3d89accf.el 250 kB/s | 27 kB 00:00 2024-06-09T21:53:08.432 INFO:teuthology.orchestra.run.smithi032.stdout:(26/181): libcephfs2-debuginfo-19.0.0-2946.g3d8 6.3 MB/s | 5.4 MB 00:00 2024-06-09T21:53:08.439 INFO:teuthology.orchestra.run.smithi148.stdout:(24/181): libcephfs-devel-19.0.0-2946.g3d89accf 347 kB/s | 32 kB 00:00 2024-06-09T21:53:08.524 INFO:teuthology.orchestra.run.smithi032.stdout:(27/181): libcephsqlite-19.0.0-2946.g3d89accf.e 1.7 MB/s | 163 kB 00:00 2024-06-09T21:53:08.616 INFO:teuthology.orchestra.run.smithi032.stdout:(28/181): librados-devel-19.0.0-2946.g3d89accf. 1.4 MB/s | 127 kB 00:00 2024-06-09T21:53:08.715 INFO:teuthology.orchestra.run.smithi148.stdout:(25/181): libcephfs2-19.0.0-2946.g3d89accf.el9. 2.7 MB/s | 747 kB 00:00 2024-06-09T21:53:08.733 INFO:teuthology.orchestra.run.smithi032.stdout:(29/181): librados-devel-debuginfo-19.0.0-2946. 933 kB/s | 109 kB 00:00 2024-06-09T21:53:09.715 INFO:teuthology.orchestra.run.smithi148.stdout:(26/181): libcephfs2-debuginfo-19.0.0-2946.g3d8 5.4 MB/s | 5.4 MB 00:00 2024-06-09T21:53:09.815 INFO:teuthology.orchestra.run.smithi148.stdout:(27/181): libcephsqlite-19.0.0-2946.g3d89accf.e 1.6 MB/s | 163 kB 00:00 2024-06-09T21:53:09.916 INFO:teuthology.orchestra.run.smithi148.stdout:(28/181): librados-devel-19.0.0-2946.g3d89accf. 1.2 MB/s | 127 kB 00:00 2024-06-09T21:53:10.007 INFO:teuthology.orchestra.run.smithi148.stdout:(29/181): librados-devel-debuginfo-19.0.0-2946. 1.2 MB/s | 109 kB 00:00 2024-06-09T21:53:15.874 INFO:teuthology.orchestra.run.smithi148.stdout:(30/181): librados2-debuginfo-19.0.0-2946.g3d89 7.7 MB/s | 45 MB 00:05 2024-06-09T21:53:16.041 INFO:teuthology.orchestra.run.smithi148.stdout:(31/181): libradosstriper1-19.0.0-2946.g3d89acc 2.9 MB/s | 501 kB 00:00 2024-06-09T21:53:17.416 INFO:teuthology.orchestra.run.smithi032.stdout:(30/181): librados2-debuginfo-19.0.0-2946.g3d89 5.2 MB/s | 45 MB 00:08 2024-06-09T21:53:17.591 INFO:teuthology.orchestra.run.smithi032.stdout:(31/181): libradosstriper1-19.0.0-2946.g3d89acc 2.8 MB/s | 501 kB 00:00 2024-06-09T21:53:22.174 INFO:teuthology.orchestra.run.smithi148.stdout:(32/181): librbd1-debuginfo-19.0.0-2946.g3d89ac 8.5 MB/s | 52 MB 00:06 2024-06-09T21:53:22.957 INFO:teuthology.orchestra.run.smithi148.stdout:(33/181): librgw2-19.0.0-2946.g3d89accf.el9.x86 6.8 MB/s | 5.3 MB 00:00 2024-06-09T21:53:23.049 INFO:teuthology.orchestra.run.smithi148.stdout:(34/181): python3-ceph-argparse-19.0.0-2946.g3d 493 kB/s | 45 kB 00:00 2024-06-09T21:53:23.142 INFO:teuthology.orchestra.run.smithi148.stdout:(35/181): python3-ceph-common-19.0.0-2946.g3d89 1.4 MB/s | 135 kB 00:00 2024-06-09T21:53:23.258 INFO:teuthology.orchestra.run.smithi148.stdout:(36/181): python3-cephfs-19.0.0-2946.g3d89accf. 1.4 MB/s | 162 kB 00:00 2024-06-09T21:53:23.408 INFO:teuthology.orchestra.run.smithi148.stdout:(37/181): python3-cephfs-debuginfo-19.0.0-2946. 4.0 MB/s | 615 kB 00:00 2024-06-09T21:53:23.517 INFO:teuthology.orchestra.run.smithi148.stdout:(38/181): python3-rados-19.0.0-2946.g3d89accf.e 2.9 MB/s | 321 kB 00:00 2024-06-09T21:53:24.159 INFO:teuthology.orchestra.run.smithi148.stdout:(39/181): python3-rados-debuginfo-19.0.0-2946.g 1.2 MB/s | 771 kB 00:00 2024-06-09T21:53:24.292 INFO:teuthology.orchestra.run.smithi148.stdout:(40/181): python3-rbd-19.0.0-2946.g3d89accf.el9 2.2 MB/s | 300 kB 00:00 2024-06-09T21:53:24.584 INFO:teuthology.orchestra.run.smithi148.stdout:(41/181): python3-rbd-debuginfo-19.0.0-2946.g3d 3.9 MB/s | 1.1 MB 00:00 2024-06-09T21:53:24.685 INFO:teuthology.orchestra.run.smithi148.stdout:(42/181): python3-rgw-19.0.0-2946.g3d89accf.el9 995 kB/s | 99 kB 00:00 2024-06-09T21:53:24.818 INFO:teuthology.orchestra.run.smithi148.stdout:(43/181): python3-rgw-debuginfo-19.0.0-2946.g3d 2.5 MB/s | 337 kB 00:00 2024-06-09T21:53:24.910 INFO:teuthology.orchestra.run.smithi148.stdout:(44/181): rbd-fuse-19.0.0-2946.g3d89accf.el9.x8 930 kB/s | 85 kB 00:00 2024-06-09T21:53:25.061 INFO:teuthology.orchestra.run.smithi148.stdout:(45/181): rbd-fuse-debuginfo-19.0.0-2946.g3d89a 3.4 MB/s | 516 kB 00:00 2024-06-09T21:53:25.628 INFO:teuthology.orchestra.run.smithi148.stdout:(46/181): rbd-mirror-19.0.0-2946.g3d89accf.el9. 5.5 MB/s | 3.1 MB 00:00 2024-06-09T21:53:27.691 INFO:teuthology.orchestra.run.smithi032.stdout:(32/181): librbd1-debuginfo-19.0.0-2946.g3d89ac 5.1 MB/s | 52 MB 00:10 2024-06-09T21:53:27.770 INFO:teuthology.orchestra.run.smithi148.stdout:(47/181): ceph-osd-debuginfo-19.0.0-2946.g3d89a 3.9 MB/s | 288 MB 01:14 2024-06-09T21:53:27.953 INFO:teuthology.orchestra.run.smithi148.stdout:(48/181): rbd-nbd-19.0.0-2946.g3d89accf.el9.x86 933 kB/s | 171 kB 00:00 2024-06-09T21:53:28.478 INFO:teuthology.orchestra.run.smithi148.stdout:(49/181): rbd-nbd-debuginfo-19.0.0-2946.g3d89ac 2.4 MB/s | 1.3 MB 00:00 2024-06-09T21:53:28.562 INFO:teuthology.orchestra.run.smithi148.stdout:(50/181): ceph-grafana-dashboards-19.0.0-2946.g 381 kB/s | 32 kB 00:00 2024-06-09T21:53:28.654 INFO:teuthology.orchestra.run.smithi148.stdout:(51/181): ceph-mgr-cephadm-19.0.0-2946.g3d89acc 1.5 MB/s | 142 kB 00:00 2024-06-09T21:53:28.757 INFO:teuthology.orchestra.run.smithi032.stdout:(33/181): librgw2-19.0.0-2946.g3d89accf.el9.x86 5.0 MB/s | 5.3 MB 00:01 2024-06-09T21:53:28.841 INFO:teuthology.orchestra.run.smithi032.stdout:(34/181): python3-ceph-argparse-19.0.0-2946.g3d 542 kB/s | 45 kB 00:00 2024-06-09T21:53:28.916 INFO:teuthology.orchestra.run.smithi032.stdout:(35/181): python3-ceph-common-19.0.0-2946.g3d89 1.8 MB/s | 135 kB 00:00 2024-06-09T21:53:29.000 INFO:teuthology.orchestra.run.smithi032.stdout:(36/181): python3-cephfs-19.0.0-2946.g3d89accf. 1.9 MB/s | 162 kB 00:00 2024-06-09T21:53:29.150 INFO:teuthology.orchestra.run.smithi032.stdout:(37/181): python3-cephfs-debuginfo-19.0.0-2946. 4.0 MB/s | 615 kB 00:00 2024-06-09T21:53:29.250 INFO:teuthology.orchestra.run.smithi032.stdout:(38/181): python3-rados-19.0.0-2946.g3d89accf.e 3.1 MB/s | 321 kB 00:00 2024-06-09T21:53:29.476 INFO:teuthology.orchestra.run.smithi032.stdout:(39/181): python3-rados-debuginfo-19.0.0-2946.g 3.3 MB/s | 771 kB 00:00 2024-06-09T21:53:29.577 INFO:teuthology.orchestra.run.smithi032.stdout:(40/181): python3-rbd-19.0.0-2946.g3d89accf.el9 2.9 MB/s | 300 kB 00:00 2024-06-09T21:53:29.629 INFO:teuthology.orchestra.run.smithi148.stdout:(52/181): ceph-mgr-dashboard-19.0.0-2946.g3d89a 3.6 MB/s | 3.5 MB 00:00 2024-06-09T21:53:29.794 INFO:teuthology.orchestra.run.smithi032.stdout:(41/181): python3-rbd-debuginfo-19.0.0-2946.g3d 5.2 MB/s | 1.1 MB 00:00 2024-06-09T21:53:29.861 INFO:teuthology.orchestra.run.smithi032.stdout:(42/181): python3-rgw-19.0.0-2946.g3d89accf.el9 1.5 MB/s | 99 kB 00:00 2024-06-09T21:53:29.919 INFO:teuthology.orchestra.run.smithi032.stdout:(43/181): python3-rgw-debuginfo-19.0.0-2946.g3d 5.6 MB/s | 337 kB 00:00 2024-06-09T21:53:30.287 INFO:teuthology.orchestra.run.smithi032.stdout:(44/181): rbd-fuse-19.0.0-2946.g3d89accf.el9.x8 232 kB/s | 85 kB 00:00 2024-06-09T21:53:30.745 INFO:teuthology.orchestra.run.smithi032.stdout:(45/181): rbd-fuse-debuginfo-19.0.0-2946.g3d89a 1.1 MB/s | 516 kB 00:00 2024-06-09T21:53:31.687 INFO:teuthology.orchestra.run.smithi032.stdout:(46/181): rbd-mirror-19.0.0-2946.g3d89accf.el9. 3.3 MB/s | 3.1 MB 00:00 2024-06-09T21:53:31.954 INFO:teuthology.orchestra.run.smithi148.stdout:(53/181): ceph-mgr-diskprediction-local-19.0.0- 3.2 MB/s | 7.4 MB 00:02 2024-06-09T21:53:32.146 INFO:teuthology.orchestra.run.smithi148.stdout:(54/181): ceph-mgr-modules-core-19.0.0-2946.g3d 1.3 MB/s | 249 kB 00:00 2024-06-09T21:53:32.263 INFO:teuthology.orchestra.run.smithi148.stdout:(55/181): ceph-mgr-rook-19.0.0-2946.g3d89accf.e 423 kB/s | 49 kB 00:00 2024-06-09T21:53:32.372 INFO:teuthology.orchestra.run.smithi148.stdout:(56/181): ceph-prometheus-alerts-19.0.0-2946.g3 152 kB/s | 16 kB 00:00 2024-06-09T21:53:32.555 INFO:teuthology.orchestra.run.smithi148.stdout:(57/181): ceph-volume-19.0.0-2946.g3d89accf.el9 1.4 MB/s | 264 kB 00:00 2024-06-09T21:53:32.905 INFO:teuthology.orchestra.run.smithi148.stdout:(58/181): cephadm-19.0.0-2946.g3d89accf.el9.noa 2.1 MB/s | 763 kB 00:00 2024-06-09T21:53:33.097 INFO:teuthology.orchestra.run.smithi148.stdout:(59/181): cephfs-top-19.0.0-2946.g3d89accf.el9. 95 kB/s | 18 kB 00:00 2024-06-09T21:53:33.134 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:33.248 INFO:teuthology.orchestra.run.smithi148.stdout:(60/181): fuse-2.9.9-15.el9.x86_64.rpm 536 kB/s | 80 kB 00:00 2024-06-09T21:53:33.249 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:33.373 INFO:teuthology.orchestra.run.smithi148.stdout:(61/181): libgfortran-11.4.1-3.el9.x86_64.rpm 6.3 MB/s | 803 kB 00:00 2024-06-09T21:53:33.374 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:33.431 INFO:teuthology.orchestra.run.smithi148.stdout:(62/181): libquadmath-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 195 kB 00:00 2024-06-09T21:53:33.490 INFO:teuthology.orchestra.run.smithi148.stdout:(63/181): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.2 MB/s | 253 kB 00:00 2024-06-09T21:53:33.574 INFO:teuthology.orchestra.run.smithi148.stdout:(64/181): python3-cryptography-36.0.1-4.el9.x86 15 MB/s | 1.2 MB 00:00 2024-06-09T21:53:33.624 INFO:teuthology.orchestra.run.smithi148.stdout:(65/181): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-06-09T21:53:33.675 INFO:teuthology.orchestra.run.smithi148.stdout:(66/181): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-06-09T21:53:33.725 INFO:teuthology.orchestra.run.smithi148.stdout:(67/181): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-06-09T21:53:33.775 INFO:teuthology.orchestra.run.smithi148.stdout:(68/181): python3-urllib3-1.26.5-5.el9.noarch.r 4.2 MB/s | 215 kB 00:00 2024-06-09T21:53:33.776 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-09T21:53:34.501 INFO:teuthology.orchestra.run.smithi148.stdout:(69/181): automake-1.16.2-8.el9.noarch.rpm 950 kB/s | 689 kB 00:00 2024-06-09T21:53:34.501 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:34.701 INFO:teuthology.orchestra.run.smithi148.stdout:(70/181): bison-3.7.4-5.el9.x86_64.rpm 4.6 MB/s | 943 kB 00:00 2024-06-09T21:53:34.702 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:35.026 INFO:teuthology.orchestra.run.smithi148.stdout:(71/181): boost-program-options-1.75.0-8.el9.x8 329 kB/s | 107 kB 00:00 2024-06-09T21:53:35.376 INFO:teuthology.orchestra.run.smithi148.stdout:(72/181): elfutils-libelf-devel-0.191-4.el9.x86 62 kB/s | 22 kB 00:00 2024-06-09T21:53:35.719 INFO:teuthology.orchestra.run.smithi148.stdout:(73/181): flex-2.6.4-9.el9.x86_64.rpm 918 kB/s | 313 kB 00:00 2024-06-09T21:53:36.061 INFO:teuthology.orchestra.run.smithi148.stdout:(74/181): flexiblas-3.0.4-8.el9.x86_64.rpm 96 kB/s | 33 kB 00:00 2024-06-09T21:53:36.302 INFO:teuthology.orchestra.run.smithi148.stdout:(75/181): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-06-09T21:53:36.753 INFO:teuthology.orchestra.run.smithi148.stdout:(76/181): rbd-mirror-debuginfo-19.0.0-2946.g3d8 4.6 MB/s | 52 MB 00:11 2024-06-09T21:53:36.778 INFO:teuthology.orchestra.run.smithi148.stdout:(77/181): flexiblas-openblas-openmp-3.0.4-8.el9 38 kB/s | 18 kB 00:00 2024-06-09T21:53:36.853 INFO:teuthology.orchestra.run.smithi148.stdout:(78/181): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 11 kB 00:00 2024-06-09T21:53:36.878 INFO:teuthology.orchestra.run.smithi148.stdout:(79/181): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-06-09T21:53:36.954 INFO:teuthology.orchestra.run.smithi148.stdout:(80/181): librabbitmq-0.11.0-7.el9.x86_64.rpm 453 kB/s | 45 kB 00:00 2024-06-09T21:53:36.987 INFO:teuthology.orchestra.run.smithi148.stdout:(81/181): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 662 kB 00:00 2024-06-09T21:53:37.062 INFO:teuthology.orchestra.run.smithi148.stdout:(82/181): libtool-2.4.6-45.el9.x86_64.rpm 5.3 MB/s | 582 kB 00:00 2024-06-09T21:53:37.088 INFO:teuthology.orchestra.run.smithi148.stdout:(83/181): libuuid-devel-2.37.4-18.el9.x86_64.rp 253 kB/s | 25 kB 00:00 2024-06-09T21:53:37.163 INFO:teuthology.orchestra.run.smithi148.stdout:(84/181): libzstd-devel-1.5.1-2.el9.x86_64.rpm 473 kB/s | 47 kB 00:00 2024-06-09T21:53:37.188 INFO:teuthology.orchestra.run.smithi148.stdout:(85/181): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2024-06-09T21:53:37.272 INFO:teuthology.orchestra.run.smithi148.stdout:(86/181): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-06-09T21:53:37.297 INFO:teuthology.orchestra.run.smithi148.stdout:(87/181): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 35 kB 00:00 2024-06-09T21:53:37.556 INFO:teuthology.orchestra.run.smithi148.stdout:(88/181): openblas-openmp-0.3.21-2.el9.x86_64.r 17 MB/s | 4.7 MB 00:00 2024-06-09T21:53:37.698 INFO:teuthology.orchestra.run.smithi148.stdout:(89/181): openssl-devel-3.2.1-1.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-06-09T21:53:37.740 INFO:teuthology.orchestra.run.smithi148.stdout:(90/181): postgresql-13.14-1.el9.x86_64.rpm 8.8 MB/s | 1.6 MB 00:00 2024-06-09T21:53:37.815 INFO:teuthology.orchestra.run.smithi148.stdout:(91/181): postgresql-contrib-13.14-1.el9.x86_64 7.8 MB/s | 911 kB 00:00 2024-06-09T21:53:37.840 INFO:teuthology.orchestra.run.smithi148.stdout:(92/181): postgresql-private-libs-13.14-1.el9.x 1.3 MB/s | 133 kB 00:00 2024-06-09T21:53:38.082 INFO:teuthology.orchestra.run.smithi148.stdout:(93/181): postgresql-server-13.14-1.el9.x86_64. 22 MB/s | 5.8 MB 00:00 2024-06-09T21:53:38.116 INFO:teuthology.orchestra.run.smithi148.stdout:(94/181): protobuf-3.14.0-13.el9.x86_64.rpm 3.7 MB/s | 1.0 MB 00:00 2024-06-09T21:53:38.183 INFO:teuthology.orchestra.run.smithi148.stdout:(95/181): python3-attrs-20.3.0-7.el9.noarch.rpm 888 kB/s | 87 kB 00:00 2024-06-09T21:53:38.309 INFO:teuthology.orchestra.run.smithi148.stdout:(96/181): python3-colorama-0.4.6-3.el9.noarch.r 489 kB/s | 61 kB 00:00 2024-06-09T21:53:38.384 INFO:teuthology.orchestra.run.smithi148.stdout:(97/181): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 6.0 MB 00:00 2024-06-09T21:53:38.417 INFO:teuthology.orchestra.run.smithi148.stdout:(98/181): python3-devel-3.9.18-3.el9.x86_64.rpm 2.2 MB/s | 244 kB 00:00 2024-06-09T21:53:38.484 INFO:teuthology.orchestra.run.smithi148.stdout:(99/181): python3-jinja2-2.11.3-6.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-06-09T21:53:38.518 INFO:teuthology.orchestra.run.smithi148.stdout:(100/181): python3-jsonpatch-1.21-16.el9.noarch 255 kB/s | 26 kB 00:00 2024-06-09T21:53:38.585 INFO:teuthology.orchestra.run.smithi148.stdout:(101/181): python3-jsonpointer-2.0-4.el9.noarch 191 kB/s | 19 kB 00:00 2024-06-09T21:53:38.618 INFO:teuthology.orchestra.run.smithi148.stdout:(102/181): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-06-09T21:53:38.686 INFO:teuthology.orchestra.run.smithi148.stdout:(103/181): python3-markupsafe-1.1.1-12.el9.x86_ 346 kB/s | 35 kB 00:00 2024-06-09T21:53:38.869 INFO:teuthology.orchestra.run.smithi148.stdout:(104/181): python3-numpy-1.20.1-5.el9.x86_64.rp 20 MB/s | 5.1 MB 00:00 2024-06-09T21:53:38.894 INFO:teuthology.orchestra.run.smithi148.stdout:(105/181): python3-numpy-f2py-1.20.1-5.el9.x86_ 1.2 MB/s | 250 kB 00:00 2024-06-09T21:53:38.986 INFO:teuthology.orchestra.run.smithi148.stdout:(106/181): python3-oauthlib-3.1.1-5.el9.noarch. 1.9 MB/s | 222 kB 00:00 2024-06-09T21:53:39.011 INFO:teuthology.orchestra.run.smithi148.stdout:(107/181): python3-packaging-20.9-5.el9.noarch. 659 kB/s | 77 kB 00:00 2024-06-09T21:53:39.087 INFO:teuthology.orchestra.run.smithi148.stdout:(108/181): python3-prettytable-0.7.2-27.el9.noa 422 kB/s | 42 kB 00:00 2024-06-09T21:53:39.112 INFO:teuthology.orchestra.run.smithi148.stdout:(109/181): python3-protobuf-3.14.0-13.el9.noarc 2.6 MB/s | 269 kB 00:00 2024-06-09T21:53:39.188 INFO:teuthology.orchestra.run.smithi148.stdout:(110/181): python3-pyasn1-0.4.8-6.el9.noarch.rp 1.5 MB/s | 159 kB 00:00 2024-06-09T21:53:39.212 INFO:teuthology.orchestra.run.smithi148.stdout:(111/181): python3-pyasn1-modules-0.4.8-6.el9.n 2.7 MB/s | 279 kB 00:00 2024-06-09T21:53:39.288 INFO:teuthology.orchestra.run.smithi148.stdout:(112/181): python3-pytz-2021.1-5.el9.noarch.rpm 509 kB/s | 51 kB 00:00 2024-06-09T21:53:39.313 INFO:teuthology.orchestra.run.smithi148.stdout:(113/181): python3-requests-oauthlib-1.3.0-12.e 534 kB/s | 54 kB 00:00 2024-06-09T21:53:39.497 INFO:teuthology.orchestra.run.smithi148.stdout:(114/181): python3-toml-0.10.2-6.el9.noarch.rpm 227 kB/s | 42 kB 00:00 2024-06-09T21:53:39.689 INFO:teuthology.orchestra.run.smithi148.stdout:(115/181): python3-wcwidth-0.2.5-8.el9.noarch.r 233 kB/s | 44 kB 00:00 2024-06-09T21:53:39.804 INFO:teuthology.orchestra.run.smithi032.stdout:(47/181): rbd-mirror-debuginfo-19.0.0-2946.g3d8 6.4 MB/s | 52 MB 00:08 2024-06-09T21:53:39.903 INFO:teuthology.orchestra.run.smithi032.stdout:(48/181): rbd-nbd-19.0.0-2946.g3d89accf.el9.x86 1.7 MB/s | 171 kB 00:00 2024-06-09T21:53:39.931 INFO:teuthology.orchestra.run.smithi148.stdout:(116/181): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-06-09T21:53:39.956 INFO:teuthology.orchestra.run.smithi148.stdout:(117/181): qatlib-24.02.0-1.el9.x86_64.rpm 834 kB/s | 221 kB 00:00 2024-06-09T21:53:40.031 INFO:teuthology.orchestra.run.smithi148.stdout:(118/181): qatlib-service-24.02.0-1.el9.x86_64. 363 kB/s | 36 kB 00:00 2024-06-09T21:53:40.056 INFO:teuthology.orchestra.run.smithi148.stdout:(119/181): qatzip-libs-1.2.0-1.el9.x86_64.rpm 474 kB/s | 47 kB 00:00 2024-06-09T21:53:40.140 INFO:teuthology.orchestra.run.smithi148.stdout:(120/181): socat-1.7.4.1-5.el9.x86_64.rpm 2.7 MB/s | 305 kB 00:00 2024-06-09T21:53:40.166 INFO:teuthology.orchestra.run.smithi148.stdout:(121/181): uuid-1.6.2-55.el9.x86_64.rpm 532 kB/s | 58 kB 00:00 2024-06-09T21:53:40.241 INFO:teuthology.orchestra.run.smithi148.stdout:(122/181): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 934 kB/s | 94 kB 00:00 2024-06-09T21:53:40.266 INFO:teuthology.orchestra.run.smithi148.stdout:(123/181): xmlstarlet-1.6.1-20.el9.x86_64.rpm 635 kB/s | 64 kB 00:00 2024-06-09T21:53:40.303 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:40.304 INFO:teuthology.orchestra.run.smithi032.stdout:(49/181): rbd-nbd-debuginfo-19.0.0-2946.g3d89ac 3.2 MB/s | 1.3 MB 00:00 2024-06-09T21:53:40.350 INFO:teuthology.orchestra.run.smithi148.stdout:(124/181): zlib-devel-1.2.11-41.el9.x86_64.rpm 413 kB/s | 45 kB 00:00 2024-06-09T21:53:40.350 INFO:teuthology.orchestra.run.smithi148.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:40.387 INFO:teuthology.orchestra.run.smithi032.stdout:(50/181): ceph-grafana-dashboards-19.0.0-2946.g 382 kB/s | 32 kB 00:00 2024-06-09T21:53:40.400 INFO:teuthology.orchestra.run.smithi148.stdout:(125/181): lua-devel-5.4.4-4.el9.x86_64.rpm 166 kB/s | 22 kB 00:00 2024-06-09T21:53:40.442 INFO:teuthology.orchestra.run.smithi148.stdout:(126/181): abseil-cpp-20211102.0-3.el9.x86_64.r 13 MB/s | 548 kB 00:00 2024-06-09T21:53:40.463 INFO:teuthology.orchestra.run.smithi032.stdout:(51/181): ceph-mgr-cephadm-19.0.0-2946.g3d89acc 1.9 MB/s | 142 kB 00:00 2024-06-09T21:53:40.467 INFO:teuthology.orchestra.run.smithi148.stdout:(127/181): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-06-09T21:53:40.493 INFO:teuthology.orchestra.run.smithi148.stdout:(128/181): grpc-data-1.46.7-10.el9.noarch.rpm 779 kB/s | 19 kB 00:00 2024-06-09T21:53:40.534 INFO:teuthology.orchestra.run.smithi148.stdout:(129/181): protobuf-compiler-3.14.0-13.el9.x86_ 4.6 MB/s | 863 kB 00:00 2024-06-09T21:53:40.601 INFO:teuthology.orchestra.run.smithi148.stdout:(130/181): libarrow-doc-9.0.0-11.el9.noarch.rpm 390 kB/s | 26 kB 00:00 2024-06-09T21:53:40.635 INFO:teuthology.orchestra.run.smithi148.stdout:(131/181): liboath-2.6.7-2.el9.x86_64.rpm 1.4 MB/s | 49 kB 00:00 2024-06-09T21:53:40.660 INFO:teuthology.orchestra.run.smithi148.stdout:(132/181): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-06-09T21:53:40.736 INFO:teuthology.orchestra.run.smithi148.stdout:(133/181): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-06-09T21:53:40.761 INFO:teuthology.orchestra.run.smithi148.stdout:(134/181): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-06-09T21:53:40.803 INFO:teuthology.orchestra.run.smithi148.stdout:(135/181): parquet-libs-9.0.0-11.el9.x86_64.rpm 12 MB/s | 839 kB 00:00 2024-06-09T21:53:40.820 INFO:teuthology.orchestra.run.smithi148.stdout:(136/181): python3-bcrypt-3.2.2-1.el9.x86_64.rp 2.6 MB/s | 43 kB 00:00 2024-06-09T21:53:40.837 INFO:teuthology.orchestra.run.smithi148.stdout:(137/181): python3-cachetools-4.2.4-1.el9.noarc 1.9 MB/s | 32 kB 00:00 2024-06-09T21:53:40.854 INFO:teuthology.orchestra.run.smithi148.stdout:(138/181): python3-certifi-2021.10.8-2.el9.noar 908 kB/s | 15 kB 00:00 2024-06-09T21:53:40.887 INFO:teuthology.orchestra.run.smithi148.stdout:(139/181): python3-asyncssh-2.13.2-5.el9.noarch 4.2 MB/s | 548 kB 00:00 2024-06-09T21:53:40.913 INFO:teuthology.orchestra.run.smithi148.stdout:(140/181): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-06-09T21:53:40.938 INFO:teuthology.orchestra.run.smithi148.stdout:(141/181): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-06-09T21:53:40.963 INFO:teuthology.orchestra.run.smithi148.stdout:(142/181): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-06-09T21:53:40.988 INFO:teuthology.orchestra.run.smithi148.stdout:(143/181): python3-google-auth-2.29.0-2.el9.noa 4.0 MB/s | 205 kB 00:00 2024-06-09T21:53:41.022 INFO:teuthology.orchestra.run.smithi148.stdout:(144/181): python3-grpcio-tools-1.46.7-10.el9.x 4.2 MB/s | 144 kB 00:00 2024-06-09T21:53:41.047 INFO:teuthology.orchestra.run.smithi148.stdout:(145/181): python3-jaraco-8.2.1-3.el9.noarch.rp 431 kB/s | 11 kB 00:00 2024-06-09T21:53:41.063 INFO:teuthology.orchestra.run.smithi032.stdout:(52/181): ceph-mgr-dashboard-19.0.0-2946.g3d89a 5.9 MB/s | 3.5 MB 00:00 2024-06-09T21:53:41.073 INFO:teuthology.orchestra.run.smithi148.stdout:(146/181): python3-jaraco-classes-3.2.1-5.el9.n 710 kB/s | 18 kB 00:00 2024-06-09T21:53:41.098 INFO:teuthology.orchestra.run.smithi148.stdout:(147/181): python3-jaraco-collections-3.0.0-8.e 933 kB/s | 23 kB 00:00 2024-06-09T21:53:41.123 INFO:teuthology.orchestra.run.smithi148.stdout:(148/181): python3-jaraco-functools-3.5.0-2.el9 775 kB/s | 19 kB 00:00 2024-06-09T21:53:41.148 INFO:teuthology.orchestra.run.smithi148.stdout:(149/181): python3-jaraco-text-3.2.0-6.el9.noar 795 kB/s | 20 kB 00:00 2024-06-09T21:53:41.199 INFO:teuthology.orchestra.run.smithi148.stdout:(150/181): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-06-09T21:53:41.225 INFO:teuthology.orchestra.run.smithi148.stdout:(151/181): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-09T21:53:41.250 INFO:teuthology.orchestra.run.smithi148.stdout:(152/181): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-06-09T21:53:41.275 INFO:teuthology.orchestra.run.smithi148.stdout:(153/181): python3-natsort-7.1.1-5.el9.noarch.r 2.3 MB/s | 58 kB 00:00 2024-06-09T21:53:41.317 INFO:teuthology.orchestra.run.smithi148.stdout:(154/181): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-06-09T21:53:41.342 INFO:teuthology.orchestra.run.smithi148.stdout:(155/181): python3-portend-3.1.0-2.el9.noarch.r 657 kB/s | 16 kB 00:00 2024-06-09T21:53:41.368 INFO:teuthology.orchestra.run.smithi148.stdout:(156/181): python3-pyOpenSSL-21.0.0-1.el9.noarc 3.5 MB/s | 90 kB 00:00 2024-06-09T21:53:41.401 INFO:teuthology.orchestra.run.smithi148.stdout:(157/181): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-06-09T21:53:41.426 INFO:teuthology.orchestra.run.smithi148.stdout:(158/181): python3-pyperclip-1.8.2-2.el9.noarch 483 kB/s | 28 kB 00:00 2024-06-09T21:53:41.452 INFO:teuthology.orchestra.run.smithi148.stdout:(159/181): python3-repoze-lru-0.7-16.el9.noarch 611 kB/s | 31 kB 00:00 2024-06-09T21:53:41.477 INFO:teuthology.orchestra.run.smithi148.stdout:(160/181): python3-rsa-4.9-2.el9.noarch.rpm 2.3 MB/s | 59 kB 00:00 2024-06-09T21:53:41.502 INFO:teuthology.orchestra.run.smithi148.stdout:(161/181): python3-tempora-5.0.0-2.el9.noarch.r 1.4 MB/s | 36 kB 00:00 2024-06-09T21:53:41.528 INFO:teuthology.orchestra.run.smithi148.stdout:(162/181): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-06-09T21:53:41.553 INFO:teuthology.orchestra.run.smithi148.stdout:(163/181): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-06-09T21:53:41.578 INFO:teuthology.orchestra.run.smithi148.stdout:(164/181): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-06-09T21:53:41.603 INFO:teuthology.orchestra.run.smithi148.stdout:(165/181): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-06-09T21:53:41.629 INFO:teuthology.orchestra.run.smithi148.stdout:(166/181): python3-zc-lockfile-2.0-10.el9.noarc 799 kB/s | 20 kB 00:00 2024-06-09T21:53:41.654 INFO:teuthology.orchestra.run.smithi148.stdout:(167/181): re2-20211101-3.el9.x86_64.rpm 7.6 MB/s | 192 kB 00:00 2024-06-09T21:53:41.679 INFO:teuthology.orchestra.run.smithi148.stdout:(168/181): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-06-09T21:53:41.738 INFO:teuthology.orchestra.run.smithi148.stdout:(169/181): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-06-09T21:53:42.522 INFO:teuthology.orchestra.run.smithi148.stdout:(170/181): librados2-19.0.0-2946.g3d89accf.el9. 4.0 MB/s | 3.4 MB 00:00 2024-06-09T21:53:42.664 INFO:teuthology.orchestra.run.smithi148.stdout:(171/181): librbd1-19.0.0-2946.g3d89accf.el9.x8 3.4 MB/s | 3.1 MB 00:00 2024-06-09T21:53:42.788 INFO:teuthology.orchestra.run.smithi032.stdout:(53/181): ceph-osd-debuginfo-19.0.0-2946.g3d89a 3.5 MB/s | 288 MB 01:23 2024-06-09T21:53:42.814 INFO:teuthology.orchestra.run.smithi148.stdout:(172/181): NetworkManager-1.47.91-1.el9.x86_64. 7.9 MB/s | 2.3 MB 00:00 2024-06-09T21:53:42.864 INFO:teuthology.orchestra.run.smithi148.stdout:(173/181): NetworkManager-team-1.47.91-1.el9.x8 832 kB/s | 41 kB 00:00 2024-06-09T21:53:42.914 INFO:teuthology.orchestra.run.smithi032.stdout:(54/181): ceph-mgr-diskprediction-local-19.0.0- 4.0 MB/s | 7.4 MB 00:01 2024-06-09T21:53:42.914 INFO:teuthology.orchestra.run.smithi148.stdout:(174/181): NetworkManager-tui-1.47.91-1.el9.x86 4.9 MB/s | 248 kB 00:00 2024-06-09T21:53:42.964 INFO:teuthology.orchestra.run.smithi032.stdout:(55/181): ceph-mgr-rook-19.0.0-2946.g3d89accf.e 995 kB/s | 49 kB 00:00 2024-06-09T21:53:42.973 INFO:teuthology.orchestra.run.smithi148.stdout:(175/181): NetworkManager-libnm-1.47.91-1.el9.x 5.8 MB/s | 1.8 MB 00:00 2024-06-09T21:53:42.998 INFO:teuthology.orchestra.run.smithi032.stdout:(56/181): ceph-prometheus-alerts-19.0.0-2946.g3 492 kB/s | 16 kB 00:00 2024-06-09T21:53:42.998 INFO:teuthology.orchestra.run.smithi148.stdout:(176/181): elfutils-debuginfod-client-0.191-4.e 439 kB/s | 37 kB 00:00 2024-06-09T21:53:43.023 INFO:teuthology.orchestra.run.smithi032.stdout:(57/181): ceph-mgr-modules-core-19.0.0-2946.g3d 1.0 MB/s | 249 kB 00:00 2024-06-09T21:53:43.023 INFO:teuthology.orchestra.run.smithi148.stdout:(177/181): elfutils-libelf-0.191-4.el9.x86_64.r 4.0 MB/s | 207 kB 00:00 2024-06-09T21:53:43.049 INFO:teuthology.orchestra.run.smithi148.stdout:(178/181): elfutils-libs-0.191-4.el9.x86_64.rpm 5.1 MB/s | 260 kB 00:00 2024-06-09T21:53:43.106 INFO:teuthology.orchestra.run.smithi032.stdout:(58/181): ceph-volume-19.0.0-2946.g3d89accf.el9 2.4 MB/s | 264 kB 00:00 2024-06-09T21:53:43.124 INFO:teuthology.orchestra.run.smithi148.stdout:(179/181): openssl-3.2.1-1.el9.x86_64.rpm 13 MB/s | 1.3 MB 00:00 2024-06-09T21:53:43.157 INFO:teuthology.orchestra.run.smithi032.stdout:(59/181): cephfs-top-19.0.0-2946.g3d89accf.el9. 365 kB/s | 18 kB 00:00 2024-06-09T21:53:43.174 INFO:teuthology.orchestra.run.smithi148.stdout:(180/181): openssl-libs-3.2.1-1.el9.x86_64.rpm 19 MB/s | 2.4 MB 00:00 2024-06-09T21:53:43.224 INFO:teuthology.orchestra.run.smithi032.stdout:(60/181): cephadm-19.0.0-2946.g3d89accf.el9.noa 3.7 MB/s | 763 kB 00:00 2024-06-09T21:53:43.227 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] fuse-2.9.9-15.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/fuse-2.9.9-15.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:43.264 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] libgfortran-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libgfortran-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:43.441 INFO:teuthology.orchestra.run.smithi032.stdout:(61/181): fuse-2.9.9-15.el9.x86_64.rpm 284 kB/s | 80 kB 00:00 2024-06-09T21:53:43.441 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] libquadmath-11.4.1-3.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/BaseOS/x86_64/os/Packages/libquadmath-11.4.1-3.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:43.532 INFO:teuthology.orchestra.run.smithi032.stdout:(62/181): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-06-09T21:53:43.608 INFO:teuthology.orchestra.run.smithi032.stdout:(63/181): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-06-09T21:53:43.633 INFO:teuthology.orchestra.run.smithi032.stdout:(64/181): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 253 kB 00:00 2024-06-09T21:53:43.692 INFO:teuthology.orchestra.run.smithi032.stdout:(65/181): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-06-09T21:53:43.751 INFO:teuthology.orchestra.run.smithi032.stdout:(66/181): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-06-09T21:53:43.801 INFO:teuthology.orchestra.run.smithi032.stdout:(67/181): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-06-09T21:53:43.860 INFO:teuthology.orchestra.run.smithi032.stdout:(68/181): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-06-09T21:53:43.861 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] automake-1.16.2-8.el9.noarch.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/automake-1.16.2-8.el9.noarch.rpm (IP: 8.43.85.72) 2024-06-09T21:53:44.002 INFO:teuthology.orchestra.run.smithi032.stdout:(69/181): python3-cryptography-36.0.1-4.el9.x86 3.1 MB/s | 1.2 MB 00:00 2024-06-09T21:53:44.002 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] bison-3.7.4-5.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/bison-3.7.4-5.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:44.119 INFO:teuthology.orchestra.run.smithi032.stdout:(70/181): automake-1.16.2-8.el9.noarch.rpm 2.6 MB/s | 689 kB 00:00 2024-06-09T21:53:44.120 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] boost-program-options-1.75.0-8.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/AppStream/x86_64/os/Packages/boost-program-options-1.75.0-8.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:44.177 INFO:teuthology.orchestra.run.smithi032.stdout:(71/181): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-06-09T21:53:44.228 INFO:teuthology.orchestra.run.smithi032.stdout:(72/181): elfutils-libelf-devel-0.191-4.el9.x86 434 kB/s | 22 kB 00:00 2024-06-09T21:53:44.297 INFO:teuthology.orchestra.run.smithi032.stdout:(73/181): bison-3.7.4-5.el9.x86_64.rpm 3.1 MB/s | 943 kB 00:00 2024-06-09T21:53:44.320 INFO:teuthology.orchestra.run.smithi032.stdout:(74/181): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-06-09T21:53:44.354 INFO:teuthology.orchestra.run.smithi032.stdout:(75/181): flexiblas-3.0.4-8.el9.x86_64.rpm 559 kB/s | 33 kB 00:00 2024-06-09T21:53:44.412 INFO:teuthology.orchestra.run.smithi032.stdout:(76/181): flexiblas-openblas-openmp-3.0.4-8.el9 312 kB/s | 18 kB 00:00 2024-06-09T21:53:44.463 INFO:teuthology.orchestra.run.smithi032.stdout:(77/181): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-06-09T21:53:44.522 INFO:teuthology.orchestra.run.smithi032.stdout:(78/181): flexiblas-netlib-3.0.4-8.el9.x86_64.r 15 MB/s | 3.0 MB 00:00 2024-06-09T21:53:44.547 INFO:teuthology.orchestra.run.smithi032.stdout:(79/181): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 160 kB 00:00 2024-06-09T21:53:44.572 INFO:teuthology.orchestra.run.smithi032.stdout:(80/181): librabbitmq-0.11.0-7.el9.x86_64.rpm 901 kB/s | 45 kB 00:00 2024-06-09T21:53:44.631 INFO:teuthology.orchestra.run.smithi032.stdout:(81/181): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 582 kB 00:00 2024-06-09T21:53:44.664 INFO:teuthology.orchestra.run.smithi032.stdout:(82/181): librdkafka-1.6.1-102.el9.x86_64.rpm 5.5 MB/s | 662 kB 00:00 2024-06-09T21:53:44.689 INFO:teuthology.orchestra.run.smithi032.stdout:(83/181): libuuid-devel-2.37.4-18.el9.x86_64.rp 433 kB/s | 25 kB 00:00 2024-06-09T21:53:44.723 INFO:teuthology.orchestra.run.smithi032.stdout:(84/181): libzstd-devel-1.5.1-2.el9.x86_64.rpm 810 kB/s | 47 kB 00:00 2024-06-09T21:53:44.749 INFO:teuthology.orchestra.run.smithi032.stdout:(85/181): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-06-09T21:53:44.782 INFO:teuthology.orchestra.run.smithi032.stdout:(86/181): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 188 kB 00:00 2024-06-09T21:53:44.807 INFO:teuthology.orchestra.run.smithi032.stdout:(87/181): openblas-0.3.21-2.el9.x86_64.rpm 601 kB/s | 35 kB 00:00 2024-06-09T21:53:45.050 INFO:teuthology.orchestra.run.smithi032.stdout:(88/181): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-06-09T21:53:45.116 INFO:teuthology.orchestra.run.smithi032.stdout:(89/181): openssl-devel-3.2.1-1.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-06-09T21:53:45.175 INFO:teuthology.orchestra.run.smithi032.stdout:(90/181): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-06-09T21:53:45.209 INFO:teuthology.orchestra.run.smithi032.stdout:(91/181): postgresql-contrib-13.14-1.el9.x86_64 9.7 MB/s | 911 kB 00:00 2024-06-09T21:53:45.226 INFO:teuthology.orchestra.run.smithi032.stdout:(92/181): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-06-09T21:53:45.301 INFO:teuthology.orchestra.run.smithi032.stdout:(93/181): protobuf-3.14.0-13.el9.x86_64.rpm 13 MB/s | 1.0 MB 00:00 2024-06-09T21:53:45.360 INFO:teuthology.orchestra.run.smithi032.stdout:(94/181): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-06-09T21:53:45.519 INFO:teuthology.orchestra.run.smithi032.stdout:(95/181): postgresql-server-13.14-1.el9.x86_64. 19 MB/s | 5.8 MB 00:00 2024-06-09T21:53:45.585 INFO:teuthology.orchestra.run.smithi032.stdout:(96/181): python3-colorama-0.4.6-3.el9.noarch.r 922 kB/s | 61 kB 00:00 2024-06-09T21:53:45.686 INFO:teuthology.orchestra.run.smithi032.stdout:(97/181): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-06-09T21:53:45.711 INFO:teuthology.orchestra.run.smithi032.stdout:(98/181): python3-devel-3.9.18-3.el9.x86_64.rpm 1.9 MB/s | 244 kB 00:00 2024-06-09T21:53:45.761 INFO:teuthology.orchestra.run.smithi032.stdout:(99/181): python3-jinja2-2.11.3-6.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-06-09T21:53:45.787 INFO:teuthology.orchestra.run.smithi032.stdout:(100/181): python3-jsonpatch-1.21-16.el9.noarch 340 kB/s | 26 kB 00:00 2024-06-09T21:53:45.812 INFO:teuthology.orchestra.run.smithi032.stdout:(101/181): python3-jsonpointer-2.0-4.el9.noarch 382 kB/s | 19 kB 00:00 2024-06-09T21:53:45.845 INFO:teuthology.orchestra.run.smithi032.stdout:(102/181): python3-mako-1.1.4-6.el9.noarch.rpm 2.9 MB/s | 172 kB 00:00 2024-06-09T21:53:45.871 INFO:teuthology.orchestra.run.smithi032.stdout:(103/181): python3-markupsafe-1.1.1-12.el9.x86_ 590 kB/s | 35 kB 00:00 2024-06-09T21:53:45.930 INFO:teuthology.orchestra.run.smithi032.stdout:(104/181): python3-numpy-f2py-1.20.1-5.el9.x86_ 4.2 MB/s | 250 kB 00:00 2024-06-09T21:53:45.988 INFO:teuthology.orchestra.run.smithi032.stdout:(105/181): python3-oauthlib-3.1.1-5.el9.noarch. 3.7 MB/s | 222 kB 00:00 2024-06-09T21:53:46.047 INFO:teuthology.orchestra.run.smithi032.stdout:(106/181): python3-packaging-20.9-5.el9.noarch. 1.3 MB/s | 77 kB 00:00 2024-06-09T21:53:46.122 INFO:teuthology.orchestra.run.smithi032.stdout:(107/181): python3-numpy-1.20.1-5.el9.x86_64.rp 18 MB/s | 5.1 MB 00:00 2024-06-09T21:53:46.147 INFO:teuthology.orchestra.run.smithi032.stdout:(108/181): python3-prettytable-0.7.2-27.el9.noa 421 kB/s | 42 kB 00:00 2024-06-09T21:53:46.181 INFO:teuthology.orchestra.run.smithi032.stdout:(109/181): python3-protobuf-3.14.0-13.el9.noarc 4.5 MB/s | 269 kB 00:00 2024-06-09T21:53:46.206 INFO:teuthology.orchestra.run.smithi032.stdout:(110/181): python3-pyasn1-0.4.8-6.el9.noarch.rp 2.6 MB/s | 159 kB 00:00 2024-06-09T21:53:46.240 INFO:teuthology.orchestra.run.smithi032.stdout:(111/181): python3-pyasn1-modules-0.4.8-6.el9.n 4.6 MB/s | 279 kB 00:00 2024-06-09T21:53:46.265 INFO:teuthology.orchestra.run.smithi032.stdout:(112/181): python3-pytz-2021.1-5.el9.noarch.rpm 867 kB/s | 51 kB 00:00 2024-06-09T21:53:46.290 INFO:teuthology.orchestra.run.smithi032.stdout:(113/181): python3-requests-oauthlib-1.3.0-12.e 1.0 MB/s | 54 kB 00:00 2024-06-09T21:53:46.341 INFO:teuthology.orchestra.run.smithi032.stdout:(114/181): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-06-09T21:53:46.400 INFO:teuthology.orchestra.run.smithi032.stdout:(115/181): python3-wcwidth-0.2.5-8.el9.noarch.r 752 kB/s | 44 kB 00:00 2024-06-09T21:53:46.467 INFO:teuthology.orchestra.run.smithi032.stdout:(116/181): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-06-09T21:53:46.525 INFO:teuthology.orchestra.run.smithi032.stdout:(117/181): qatlib-service-24.02.0-1.el9.x86_64. 615 kB/s | 36 kB 00:00 2024-06-09T21:53:46.584 INFO:teuthology.orchestra.run.smithi032.stdout:(118/181): qatzip-libs-1.2.0-1.el9.x86_64.rpm 813 kB/s | 47 kB 00:00 2024-06-09T21:53:46.660 INFO:teuthology.orchestra.run.smithi032.stdout:(119/181): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-06-09T21:53:46.710 INFO:teuthology.orchestra.run.smithi032.stdout:(120/181): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-06-09T21:53:46.902 INFO:teuthology.orchestra.run.smithi032.stdout:(121/181): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-06-09T21:53:46.927 INFO:teuthology.orchestra.run.smithi032.stdout:(122/181): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 432 kB/s | 94 kB 00:00 2024-06-09T21:53:46.944 INFO:teuthology.orchestra.run.smithi032.stdout:(123/181): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-09T21:53:46.945 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] lua-devel-5.4.4-4.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/lua-devel-5.4.4-4.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:46.978 INFO:teuthology.orchestra.run.smithi032.stdout:(124/181): zlib-devel-1.2.11-41.el9.x86_64.rpm 888 kB/s | 45 kB 00:00 2024-06-09T21:53:46.979 INFO:teuthology.orchestra.run.smithi032.stdout:[MIRROR] protobuf-compiler-3.14.0-13.el9.x86_64.rpm: Status code: 404 for https://download-cc-rdu01.fedoraproject.org/pub/centos-stream/9-stream/CRB/x86_64/os/Packages/protobuf-compiler-3.14.0-13.el9.x86_64.rpm (IP: 8.43.85.72) 2024-06-09T21:53:47.045 INFO:teuthology.orchestra.run.smithi032.stdout:(125/181): lua-devel-5.4.4-4.el9.x86_64.rpm 222 kB/s | 22 kB 00:00 2024-06-09T21:53:47.078 INFO:teuthology.orchestra.run.smithi032.stdout:(126/181): abseil-cpp-20211102.0-3.el9.x86_64.r 16 MB/s | 548 kB 00:00 2024-06-09T21:53:47.104 INFO:teuthology.orchestra.run.smithi032.stdout:(127/181): gperftools-libs-2.9.1-2.el9.x86_64.r 12 MB/s | 309 kB 00:00 2024-06-09T21:53:47.120 INFO:teuthology.orchestra.run.smithi032.stdout:(128/181): grpc-data-1.46.7-10.el9.noarch.rpm 1.1 MB/s | 19 kB 00:00 2024-06-09T21:53:47.188 INFO:teuthology.orchestra.run.smithi032.stdout:(129/181): protobuf-compiler-3.14.0-13.el9.x86_ 4.0 MB/s | 863 kB 00:00 2024-06-09T21:53:47.221 INFO:teuthology.orchestra.run.smithi032.stdout:(130/181): libarrow-doc-9.0.0-11.el9.noarch.rpm 783 kB/s | 26 kB 00:00 2024-06-09T21:53:47.246 INFO:teuthology.orchestra.run.smithi032.stdout:(131/181): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-06-09T21:53:47.272 INFO:teuthology.orchestra.run.smithi032.stdout:(132/181): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-06-09T21:53:47.297 INFO:teuthology.orchestra.run.smithi032.stdout:(133/181): luarocks-3.9.2-5.el9.noarch.rpm 6.0 MB/s | 151 kB 00:00 2024-06-09T21:53:47.364 INFO:teuthology.orchestra.run.smithi032.stdout:(134/181): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-06-09T21:53:47.398 INFO:teuthology.orchestra.run.smithi032.stdout:(135/181): python3-asyncssh-2.13.2-5.el9.noarch 16 MB/s | 548 kB 00:00 2024-06-09T21:53:47.423 INFO:teuthology.orchestra.run.smithi032.stdout:(136/181): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.7 MB/s | 43 kB 00:00 2024-06-09T21:53:47.448 INFO:teuthology.orchestra.run.smithi032.stdout:(137/181): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.4 MB/s | 839 kB 00:00 2024-06-09T21:53:47.474 INFO:teuthology.orchestra.run.smithi032.stdout:(138/181): python3-cachetools-4.2.4-1.el9.noarc 639 kB/s | 32 kB 00:00 2024-06-09T21:53:47.499 INFO:teuthology.orchestra.run.smithi032.stdout:(139/181): python3-certifi-2021.10.8-2.el9.noar 603 kB/s | 15 kB 00:00 2024-06-09T21:53:47.524 INFO:teuthology.orchestra.run.smithi032.stdout:(140/181): python3-cheroot-8.6.0-4.el9.noarch.r 3.4 MB/s | 172 kB 00:00 2024-06-09T21:53:47.550 INFO:teuthology.orchestra.run.smithi032.stdout:(141/181): python3-cherrypy-18.6.1-2.el9.noarch 7.0 MB/s | 358 kB 00:00 2024-06-09T21:53:47.575 INFO:teuthology.orchestra.run.smithi032.stdout:(142/181): python3-cmd2-2.4.2-2.el9.noarch.rpm 5.9 MB/s | 303 kB 00:00 2024-06-09T21:53:47.600 INFO:teuthology.orchestra.run.smithi032.stdout:(143/181): python3-google-auth-2.29.0-2.el9.noa 4.0 MB/s | 205 kB 00:00 2024-06-09T21:53:47.626 INFO:teuthology.orchestra.run.smithi032.stdout:(144/181): python3-grpcio-tools-1.46.7-10.el9.x 5.6 MB/s | 144 kB 00:00 2024-06-09T21:53:47.651 INFO:teuthology.orchestra.run.smithi032.stdout:(145/181): python3-jaraco-8.2.1-3.el9.noarch.rp 427 kB/s | 11 kB 00:00 2024-06-09T21:53:47.693 INFO:teuthology.orchestra.run.smithi032.stdout:(146/181): python3-jaraco-classes-3.2.1-5.el9.n 424 kB/s | 18 kB 00:00 2024-06-09T21:53:47.718 INFO:teuthology.orchestra.run.smithi032.stdout:(147/181): python3-jaraco-collections-3.0.0-8.e 925 kB/s | 23 kB 00:00 2024-06-09T21:53:47.744 INFO:teuthology.orchestra.run.smithi032.stdout:(148/181): python3-jaraco-functools-3.5.0-2.el9 776 kB/s | 19 kB 00:00 2024-06-09T21:53:47.769 INFO:teuthology.orchestra.run.smithi032.stdout:(149/181): python3-jaraco-text-3.2.0-6.el9.noar 789 kB/s | 20 kB 00:00 2024-06-09T21:53:47.819 INFO:teuthology.orchestra.run.smithi032.stdout:(150/181): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-06-09T21:53:47.845 INFO:teuthology.orchestra.run.smithi032.stdout:(151/181): python3-logutils-0.3.5-21.el9.noarch 1.8 MB/s | 46 kB 00:00 2024-06-09T21:53:47.870 INFO:teuthology.orchestra.run.smithi032.stdout:(152/181): python3-more-itertools-8.12.0-2.el9. 3.1 MB/s | 79 kB 00:00 2024-06-09T21:53:47.904 INFO:teuthology.orchestra.run.smithi032.stdout:(153/181): python3-natsort-7.1.1-5.el9.noarch.r 1.7 MB/s | 58 kB 00:00 2024-06-09T21:53:47.937 INFO:teuthology.orchestra.run.smithi032.stdout:(154/181): python3-kubernetes-26.1.0-2.el9.noar 6.1 MB/s | 1.0 MB 00:00 2024-06-09T21:53:47.963 INFO:teuthology.orchestra.run.smithi032.stdout:(155/181): python3-portend-3.1.0-2.el9.noarch.r 655 kB/s | 16 kB 00:00 2024-06-09T21:53:47.988 INFO:teuthology.orchestra.run.smithi032.stdout:(156/181): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-06-09T21:53:48.013 INFO:teuthology.orchestra.run.smithi032.stdout:(157/181): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-06-09T21:53:48.038 INFO:teuthology.orchestra.run.smithi032.stdout:(158/181): python3-pyperclip-1.8.2-2.el9.noarch 1.1 MB/s | 28 kB 00:00 2024-06-09T21:53:48.064 INFO:teuthology.orchestra.run.smithi032.stdout:(159/181): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-06-09T21:53:48.080 INFO:teuthology.orchestra.run.smithi032.stdout:(160/181): python3-rsa-4.9-2.el9.noarch.rpm 3.5 MB/s | 59 kB 00:00 2024-06-09T21:53:48.097 INFO:teuthology.orchestra.run.smithi032.stdout:(161/181): python3-tempora-5.0.0-2.el9.noarch.r 2.1 MB/s | 36 kB 00:00 2024-06-09T21:53:48.123 INFO:teuthology.orchestra.run.smithi032.stdout:(162/181): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-06-09T21:53:48.140 INFO:teuthology.orchestra.run.smithi032.stdout:(163/181): python3-typing-extensions-4.4.0-2.el 1.2 MB/s | 51 kB 00:00 2024-06-09T21:53:48.165 INFO:teuthology.orchestra.run.smithi032.stdout:(164/181): python3-websocket-client-1.2.3-2.el9 3.5 MB/s | 90 kB 00:00 2024-06-09T21:53:48.191 INFO:teuthology.orchestra.run.smithi032.stdout:(165/181): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-06-09T21:53:48.207 INFO:teuthology.orchestra.run.smithi032.stdout:(166/181): python3-zc-lockfile-2.0-10.el9.noarc 1.2 MB/s | 20 kB 00:00 2024-06-09T21:53:48.233 INFO:teuthology.orchestra.run.smithi032.stdout:(167/181): python3-werkzeug-2.0.3-3.el9.1.noarc 6.2 MB/s | 427 kB 00:00 2024-06-09T21:53:48.258 INFO:teuthology.orchestra.run.smithi032.stdout:(168/181): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-06-09T21:53:48.309 INFO:teuthology.orchestra.run.smithi032.stdout:(169/181): thrift-0.15.0-2.el9.x86_64.rpm 21 MB/s | 1.6 MB 00:00 2024-06-09T21:53:49.251 INFO:teuthology.orchestra.run.smithi032.stdout:(170/181): librbd1-19.0.0-2946.g3d89accf.el9.x8 3.3 MB/s | 3.1 MB 00:00 2024-06-09T21:53:49.951 INFO:teuthology.orchestra.run.smithi032.stdout:(171/181): NetworkManager-1.47.91-1.el9.x86_64. 3.3 MB/s | 2.3 MB 00:00 2024-06-09T21:53:50.018 INFO:teuthology.orchestra.run.smithi032.stdout:(172/181): librados2-19.0.0-2946.g3d89accf.el9. 1.9 MB/s | 3.4 MB 00:01 2024-06-09T21:53:50.176 INFO:teuthology.orchestra.run.smithi032.stdout:(173/181): NetworkManager-team-1.47.91-1.el9.x8 261 kB/s | 41 kB 00:00 2024-06-09T21:53:50.302 INFO:teuthology.orchestra.run.smithi032.stdout:(174/181): NetworkManager-tui-1.47.91-1.el9.x86 1.9 MB/s | 248 kB 00:00 2024-06-09T21:53:50.360 INFO:teuthology.orchestra.run.smithi032.stdout:(175/181): elfutils-debuginfod-client-0.191-4.e 631 kB/s | 37 kB 00:00 2024-06-09T21:53:50.419 INFO:teuthology.orchestra.run.smithi032.stdout:(176/181): elfutils-libelf-0.191-4.el9.x86_64.r 3.5 MB/s | 207 kB 00:00 2024-06-09T21:53:50.478 INFO:teuthology.orchestra.run.smithi032.stdout:(177/181): elfutils-libs-0.191-4.el9.x86_64.rpm 4.4 MB/s | 260 kB 00:00 2024-06-09T21:53:50.669 INFO:teuthology.orchestra.run.smithi032.stdout:(178/181): openssl-3.2.1-1.el9.x86_64.rpm 7.0 MB/s | 1.3 MB 00:00 2024-06-09T21:53:50.987 INFO:teuthology.orchestra.run.smithi032.stdout:(179/181): NetworkManager-libnm-1.47.91-1.el9.x 1.7 MB/s | 1.8 MB 00:01 2024-06-09T21:53:51.995 INFO:teuthology.orchestra.run.smithi032.stdout:(180/181): openssl-libs-3.2.1-1.el9.x86_64.rpm 1.8 MB/s | 2.4 MB 00:01 2024-06-09T21:54:30.983 INFO:teuthology.orchestra.run.smithi032.stdout:(181/181): ceph-test-debuginfo-19.0.0-2946.g3d8 7.4 MB/s | 692 MB 01:34 2024-06-09T21:54:30.989 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-06-09T21:54:30.989 INFO:teuthology.orchestra.run.smithi032.stdout:Total 13 MB/s | 2.2 GB 02:50 2024-06-09T21:54:32.177 INFO:teuthology.orchestra.run.smithi148.stdout:(181/181): ceph-test-debuginfo-19.0.0-2946.g3d8 7.6 MB/s | 692 MB 01:31 2024-06-09T21:54:32.183 INFO:teuthology.orchestra.run.smithi148.stdout:-------------------------------------------------------------------------------- 2024-06-09T21:54:32.183 INFO:teuthology.orchestra.run.smithi148.stdout:Total 14 MB/s | 2.2 GB 02:48 2024-06-09T21:54:32.199 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-06-09T21:54:32.381 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-06-09T21:54:32.381 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-06-09T21:54:33.353 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction check 2024-06-09T21:54:33.515 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction check succeeded. 2024-06-09T21:54:33.515 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction test 2024-06-09T21:54:47.238 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-06-09T21:54:47.239 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-06-09T21:54:47.682 INFO:teuthology.orchestra.run.smithi148.stdout:Transaction test succeeded. 2024-06-09T21:54:47.683 INFO:teuthology.orchestra.run.smithi148.stdout:Running transaction 2024-06-09T21:55:06.520 INFO:teuthology.orchestra.run.smithi148.stdout: Preparing : 1/1 2024-06-09T21:55:06.866 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-06-09T21:55:09.137 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/192 2024-06-09T21:55:09.211 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 2/192 2024-06-09T21:55:09.258 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/192 2024-06-09T21:55:09.313 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/192 2024-06-09T21:55:09.353 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/192 2024-06-09T21:55:09.378 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 6/192 2024-06-09T21:55:09.442 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 7/192 2024-06-09T21:55:09.597 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/192 2024-06-09T21:55:09.627 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 2/192 2024-06-09T21:55:09.666 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/192 2024-06-09T21:55:09.730 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 4/192 2024-06-09T21:55:09.770 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/192 2024-06-09T21:55:09.776 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 8/192 2024-06-09T21:55:09.795 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 6/192 2024-06-09T21:55:09.847 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 7/192 2024-06-09T21:55:09.888 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/192 2024-06-09T21:55:10.058 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/192 2024-06-09T21:55:10.147 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 10/192 2024-06-09T21:55:10.204 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 8/192 2024-06-09T21:55:10.244 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 10/192 2024-06-09T21:55:10.277 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 11/192 2024-06-09T21:55:10.318 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 12/192 2024-06-09T21:55:10.320 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/192 2024-06-09T21:55:10.437 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 9/192 2024-06-09T21:55:10.463 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 10/192 2024-06-09T21:55:10.505 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 13/192 2024-06-09T21:55:10.548 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 10/192 2024-06-09T21:55:10.577 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 11/192 2024-06-09T21:55:10.578 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-13.14-1.el9.x86_64 14/192 2024-06-09T21:55:10.602 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 15/192 2024-06-09T21:55:10.621 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 12/192 2024-06-09T21:55:10.690 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 15/192 2024-06-09T21:55:10.711 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 16/192 2024-06-09T21:55:10.757 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 16/192 2024-06-09T21:55:10.779 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 17/192 2024-06-09T21:55:10.805 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 17/192 2024-06-09T21:55:10.867 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-libelf-0.191-4.el9.x86_64 13/192 2024-06-09T21:55:10.932 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-13.14-1.el9.x86_64 14/192 2024-06-09T21:55:10.953 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 15/192 2024-06-09T21:55:10.997 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 18/192 2024-06-09T21:55:11.058 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 15/192 2024-06-09T21:55:11.087 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 16/192 2024-06-09T21:55:11.150 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 16/192 2024-06-09T21:55:11.168 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 18/192 2024-06-09T21:55:11.171 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 17/192 2024-06-09T21:55:11.194 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 17/192 2024-06-09T21:55:11.345 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 19/192 2024-06-09T21:55:11.383 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 18/192 2024-06-09T21:55:11.412 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 19/192 2024-06-09T21:55:11.520 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 20/192 2024-06-09T21:55:11.546 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 18/192 2024-06-09T21:55:11.605 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 20/192 2024-06-09T21:55:11.702 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 19/192 2024-06-09T21:55:11.754 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 19/192 2024-06-09T21:55:11.813 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 20/192 2024-06-09T21:55:11.871 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 20/192 2024-06-09T21:55:11.923 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/192 2024-06-09T21:55:11.947 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-1:1.47.91-1.el9.x86_64 20/192 2024-06-09T21:55:12.016 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/192 2024-06-09T21:55:12.052 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 23/192 2024-06-09T21:55:12.086 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : liboath-2.6.7-2.el9.x86_64 24/192 2024-06-09T21:55:12.154 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.47.91-1.el9.x86_64 20/192 2024-06-09T21:55:12.160 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/192 2024-06-09T21:55:12.194 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/192 2024-06-09T21:55:12.238 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/192 2024-06-09T21:55:12.271 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/192 2024-06-09T21:55:12.275 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-3.el9.x86_64 21/192 2024-06-09T21:55:12.298 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/192 2024-06-09T21:55:12.366 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/192 2024-06-09T21:55:12.370 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 22/192 2024-06-09T21:55:12.420 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/192 2024-06-09T21:55:12.420 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 23/192 2024-06-09T21:55:12.451 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/192 2024-06-09T21:55:12.455 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 24/192 2024-06-09T21:55:12.494 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/192 2024-06-09T21:55:12.531 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 25/192 2024-06-09T21:55:12.570 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/192 2024-06-09T21:55:12.571 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 26/192 2024-06-09T21:55:12.605 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 35/192 2024-06-09T21:55:12.623 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 27/192 2024-06-09T21:55:12.649 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 36/192 2024-06-09T21:55:12.655 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 28/192 2024-06-09T21:55:12.683 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 29/192 2024-06-09T21:55:12.683 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 37/192 2024-06-09T21:55:12.759 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/192 2024-06-09T21:55:12.759 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 30/192 2024-06-09T21:55:12.800 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/192 2024-06-09T21:55:12.807 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/192 2024-06-09T21:55:12.836 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/192 2024-06-09T21:55:12.875 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/192 2024-06-09T21:55:12.880 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 33/192 2024-06-09T21:55:12.967 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 34/192 2024-06-09T21:55:13.008 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 35/192 2024-06-09T21:55:13.033 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/192 2024-06-09T21:55:13.052 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 36/192 2024-06-09T21:55:13.079 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 37/192 2024-06-09T21:55:13.086 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/192 2024-06-09T21:55:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 38/192 2024-06-09T21:55:13.203 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-5.el9.noarch 39/192 2024-06-09T21:55:13.281 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 40/192 2024-06-09T21:55:13.429 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 41/192 2024-06-09T21:55:13.482 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 42/192 2024-06-09T21:55:13.609 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/192 2024-06-09T21:55:13.690 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/192 2024-06-09T21:55:13.719 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/192 2024-06-09T21:55:13.757 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/192 2024-06-09T21:55:13.805 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/192 2024-06-09T21:55:13.842 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 48/192 2024-06-09T21:55:13.884 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/192 2024-06-09T21:55:14.031 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 50/192 2024-06-09T21:55:14.049 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 43/192 2024-06-09T21:55:14.088 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 51/192 2024-06-09T21:55:14.138 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 44/192 2024-06-09T21:55:14.150 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/192 2024-06-09T21:55:14.172 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 45/192 2024-06-09T21:55:14.209 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 46/192 2024-06-09T21:55:14.223 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 53/192 2024-06-09T21:55:14.254 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 54/192 2024-06-09T21:55:14.266 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-libs-0.191-4.el9.x86_64 47/192 2024-06-09T21:55:14.291 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 55/192 2024-06-09T21:55:14.304 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 48/192 2024-06-09T21:55:14.346 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 56/192 2024-06-09T21:55:14.347 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 49/192 2024-06-09T21:55:14.382 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 57/192 2024-06-09T21:55:14.425 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 58/192 2024-06-09T21:55:14.467 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 59/192 2024-06-09T21:55:14.515 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 50/192 2024-06-09T21:55:14.566 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 51/192 2024-06-09T21:55:14.602 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 60/192 2024-06-09T21:55:14.625 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 52/192 2024-06-09T21:55:14.653 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 61/192 2024-06-09T21:55:14.694 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 62/192 2024-06-09T21:55:14.698 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 53/192 2024-06-09T21:55:14.735 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 54/192 2024-06-09T21:55:14.764 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 55/192 2024-06-09T21:55:14.799 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 63/192 2024-06-09T21:55:14.811 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 56/192 2024-06-09T21:55:14.836 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 64/192 2024-06-09T21:55:14.846 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 57/192 2024-06-09T21:55:14.884 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 65/192 2024-06-09T21:55:14.890 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 58/192 2024-06-09T21:55:14.933 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 59/192 2024-06-09T21:55:14.935 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 66/192 2024-06-09T21:55:14.960 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 67/192 2024-06-09T21:55:15.064 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 68/192 2024-06-09T21:55:15.077 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 60/192 2024-06-09T21:55:15.117 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 61/192 2024-06-09T21:55:15.159 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 62/192 2024-06-09T21:55:15.188 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 69/192 2024-06-09T21:55:15.225 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 70/192 2024-06-09T21:55:15.253 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 71/192 2024-06-09T21:55:15.258 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 63/192 2024-06-09T21:55:15.292 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 64/192 2024-06-09T21:55:15.296 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 72/192 2024-06-09T21:55:15.325 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 65/192 2024-06-09T21:55:15.335 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : uuid-1.6.2-55.el9.x86_64 73/192 2024-06-09T21:55:15.357 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 66/192 2024-06-09T21:55:15.367 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 74/192 2024-06-09T21:55:15.381 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 67/192 2024-06-09T21:55:15.396 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 75/192 2024-06-09T21:55:15.431 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 76/192 2024-06-09T21:55:15.464 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 77/192 2024-06-09T21:55:15.474 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 68/192 2024-06-09T21:55:15.505 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 78/192 2024-06-09T21:55:15.540 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 79/192 2024-06-09T21:55:15.607 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 69/192 2024-06-09T21:55:15.628 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 80/192 2024-06-09T21:55:15.639 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 70/192 2024-06-09T21:55:15.667 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 71/192 2024-06-09T21:55:15.702 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 72/192 2024-06-09T21:55:15.741 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : uuid-1.6.2-55.el9.x86_64 73/192 2024-06-09T21:55:15.781 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 74/192 2024-06-09T21:55:15.811 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 75/192 2024-06-09T21:55:15.845 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 76/192 2024-06-09T21:55:15.882 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 81/192 2024-06-09T21:55:15.895 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 77/192 2024-06-09T21:55:15.937 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 78/192 2024-06-09T21:55:15.988 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 79/192 2024-06-09T21:55:15.994 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 82/192 2024-06-09T21:55:16.041 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 83/192 2024-06-09T21:55:16.079 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 80/192 2024-06-09T21:55:16.142 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 81/192 2024-06-09T21:55:16.150 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-packaging-20.9-5.el9.noarch 84/192 2024-06-09T21:55:16.175 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 82/192 2024-06-09T21:55:16.205 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/192 2024-06-09T21:55:16.223 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 83/192 2024-06-09T21:55:16.335 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 84/192 2024-06-09T21:55:16.386 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 85/192 2024-06-09T21:55:16.931 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/192 2024-06-09T21:55:16.986 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/192 2024-06-09T21:55:17.019 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/192 2024-06-09T21:55:17.119 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/192 2024-06-09T21:55:17.175 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/192 2024-06-09T21:55:17.179 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 86/192 2024-06-09T21:55:17.220 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/192 2024-06-09T21:55:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 87/192 2024-06-09T21:55:17.257 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 88/192 2024-06-09T21:55:17.308 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/192 2024-06-09T21:55:17.339 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/192 2024-06-09T21:55:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 89/192 2024-06-09T21:55:17.414 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 90/192 2024-06-09T21:55:17.459 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-colorama-0.4.6-3.el9.noarch 91/192 2024-06-09T21:55:17.519 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/192 2024-06-09T21:55:17.550 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 92/192 2024-06-09T21:55:17.553 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/192 2024-06-09T21:55:17.593 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 93/192 2024-06-09T21:55:17.641 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/192 2024-06-09T21:55:17.779 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.21-2.el9.x86_64 94/192 2024-06-09T21:55:17.816 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 95/192 2024-06-09T21:55:17.897 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 96/192 2024-06-09T21:55:18.304 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/192 2024-06-09T21:55:18.419 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/192 2024-06-09T21:55:18.595 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 97/192 2024-06-09T21:55:18.905 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 98/192 2024-06-09T21:55:19.671 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/192 2024-06-09T21:55:19.757 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/192 2024-06-09T21:55:19.785 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/192 2024-06-09T21:55:19.818 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/192 2024-06-09T21:55:20.138 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/192 2024-06-09T21:55:20.181 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 104/192 2024-06-09T21:55:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 99/192 2024-06-09T21:55:20.271 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 104/192 2024-06-09T21:55:20.303 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 105/192 2024-06-09T21:55:20.328 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 100/192 2024-06-09T21:55:20.339 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/192 2024-06-09T21:55:20.357 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 101/192 2024-06-09T21:55:20.389 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 102/192 2024-06-09T21:55:20.717 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 103/192 2024-06-09T21:55:20.769 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 104/192 2024-06-09T21:55:20.824 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/192 2024-06-09T21:55:20.844 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 104/192 2024-06-09T21:55:20.851 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 108/192 2024-06-09T21:55:20.874 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 105/192 2024-06-09T21:55:20.905 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 108/192 2024-06-09T21:55:20.911 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 106/192 2024-06-09T21:55:20.936 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 109/192 2024-06-09T21:55:21.408 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 107/192 2024-06-09T21:55:21.447 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 108/192 2024-06-09T21:55:21.519 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 108/192 2024-06-09T21:55:21.549 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 109/192 2024-06-09T21:55:23.190 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 110/192 2024-06-09T21:55:23.224 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 110/192 2024-06-09T21:55:23.774 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 110/192 2024-06-09T21:55:23.827 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 111/192 2024-06-09T21:55:23.906 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 111/192 2024-06-09T21:55:23.906 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-09T21:55:23.906 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-09T21:55:23.906 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:23.936 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 112/192 2024-06-09T21:55:23.952 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 110/192 2024-06-09T21:55:23.987 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 110/192 2024-06-09T21:55:24.566 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 110/192 2024-06-09T21:55:24.607 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 111/192 2024-06-09T21:55:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 111/192 2024-06-09T21:55:24.698 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-09T21:55:24.698 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-09T21:55:24.698 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:24.725 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 112/192 2024-06-09T21:55:36.172 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 112/192 2024-06-09T21:55:36.172 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /sys 2024-06-09T21:55:36.173 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /proc 2024-06-09T21:55:36.173 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /mnt 2024-06-09T21:55:36.173 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /var/tmp 2024-06-09T21:55:36.173 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /home 2024-06-09T21:55:36.173 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /root 2024-06-09T21:55:36.173 INFO:teuthology.orchestra.run.smithi148.stdout:skipping the directory /tmp 2024-06-09T21:55:36.173 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:36.332 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 113/192 2024-06-09T21:55:36.347 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 113/192 2024-06-09T21:55:36.347 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:36.348 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-09T21:55:36.348 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:37.614 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 112/192 2024-06-09T21:55:37.614 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-06-09T21:55:37.614 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-06-09T21:55:37.614 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-06-09T21:55:37.615 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-06-09T21:55:37.615 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-06-09T21:55:37.615 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-06-09T21:55:37.615 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-06-09T21:55:37.615 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:37.788 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 113/192 2024-06-09T21:55:37.803 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 113/192 2024-06-09T21:55:37.803 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:37.803 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-09T21:55:37.803 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:37.835 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 114/192 2024-06-09T21:55:37.861 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 114/192 2024-06-09T21:55:37.861 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:37.861 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-09T21:55:37.861 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T21:55:37.861 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T21:55:37.862 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:38.095 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 115/192 2024-06-09T21:55:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 115/192 2024-06-09T21:55:38.118 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:38.118 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-09T21:55:38.118 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T21:55:38.118 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T21:55:38.118 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:38.563 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 116/192 2024-06-09T21:55:38.585 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 116/192 2024-06-09T21:55:38.585 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:38.585 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-09T21:55:38.586 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T21:55:38.586 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T21:55:38.586 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:38.684 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : automake-1.16.2-8.el9.noarch 117/192 2024-06-09T21:55:38.750 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/192 2024-06-09T21:55:38.806 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/192 2024-06-09T21:55:38.991 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/192 2024-06-09T21:55:39.038 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/192 2024-06-09T21:55:39.115 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/192 2024-06-09T21:55:39.211 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/192 2024-06-09T21:55:39.349 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 114/192 2024-06-09T21:55:39.349 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/192 2024-06-09T21:55:39.376 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 114/192 2024-06-09T21:55:39.376 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:39.377 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-09T21:55:39.377 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T21:55:39.377 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-09T21:55:39.377 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:39.386 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/192 2024-06-09T21:55:39.426 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/192 2024-06-09T21:55:39.454 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : fuse-2.9.9-15.el9.x86_64 127/192 2024-06-09T21:55:39.609 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 115/192 2024-06-09T21:55:39.636 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 115/192 2024-06-09T21:55:39.636 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:39.636 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-09T21:55:39.636 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T21:55:39.637 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-09T21:55:39.637 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:39.824 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 128/192 2024-06-09T21:55:39.852 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 128/192 2024-06-09T21:55:39.884 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 129/192 2024-06-09T21:55:39.947 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 130/192 2024-06-09T21:55:39.976 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 131/192 2024-06-09T21:55:40.110 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 116/192 2024-06-09T21:55:40.135 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 116/192 2024-06-09T21:55:40.135 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:40.135 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-09T21:55:40.135 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T21:55:40.135 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-09T21:55:40.135 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:40.241 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : automake-1.16.2-8.el9.noarch 117/192 2024-06-09T21:55:40.300 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 118/192 2024-06-09T21:55:40.373 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 119/192 2024-06-09T21:55:40.576 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 120/192 2024-06-09T21:55:40.627 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 121/192 2024-06-09T21:55:40.708 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/192 2024-06-09T21:55:40.806 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 123/192 2024-06-09T21:55:40.949 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 124/192 2024-06-09T21:55:41.002 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 125/192 2024-06-09T21:55:41.009 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 131/192 2024-06-09T21:55:41.034 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 126/192 2024-06-09T21:55:41.044 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 132/192 2024-06-09T21:55:41.069 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-15.el9.x86_64 127/192 2024-06-09T21:55:41.542 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 128/192 2024-06-09T21:55:41.568 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 128/192 2024-06-09T21:55:41.592 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 129/192 2024-06-09T21:55:41.656 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 130/192 2024-06-09T21:55:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 131/192 2024-06-09T21:55:42.012 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 132/192 2024-06-09T21:55:42.044 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 133/192 2024-06-09T21:55:42.147 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 133/192 2024-06-09T21:55:42.308 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 134/192 2024-06-09T21:55:42.346 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 135/192 2024-06-09T21:55:42.370 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 135/192 2024-06-09T21:55:42.370 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:42.370 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-09T21:55:42.370 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T21:55:42.370 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T21:55:42.370 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:42.414 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 136/192 2024-06-09T21:55:42.425 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 136/192 2024-06-09T21:55:42.530 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 137/192 2024-06-09T21:55:42.587 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 138/192 2024-06-09T21:55:42.773 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 131/192 2024-06-09T21:55:42.809 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 132/192 2024-06-09T21:55:43.489 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libtool-2.4.6-45.el9.x86_64 139/192 2024-06-09T21:55:43.525 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 140/192 2024-06-09T21:55:43.548 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 140/192 2024-06-09T21:55:43.548 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:43.548 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-09T21:55:43.548 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T21:55:43.549 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T21:55:43.549 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:43.601 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 141/192 2024-06-09T21:55:43.622 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 141/192 2024-06-09T21:55:43.622 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:43.623 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-09T21:55:43.623 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:43.668 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/192 2024-06-09T21:55:43.690 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/192 2024-06-09T21:55:43.690 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:43.690 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-09T21:55:43.690 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T21:55:43.690 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T21:55:43.691 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:43.826 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 132/192 2024-06-09T21:55:43.860 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 133/192 2024-06-09T21:55:43.969 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 133/192 2024-06-09T21:55:44.002 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 143/192 2024-06-09T21:55:44.024 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 143/192 2024-06-09T21:55:44.024 INFO:teuthology.orchestra.run.smithi148.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:44.024 INFO:teuthology.orchestra.run.smithi148.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-09T21:55:44.025 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T21:55:44.025 INFO:teuthology.orchestra.run.smithi148.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T21:55:44.025 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:55:44.131 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 134/192 2024-06-09T21:55:44.154 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 135/192 2024-06-09T21:55:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 135/192 2024-06-09T21:55:44.178 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:44.178 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-09T21:55:44.178 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T21:55:44.178 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-09T21:55:44.178 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:44.229 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 136/192 2024-06-09T21:55:44.241 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 136/192 2024-06-09T21:55:44.342 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 137/192 2024-06-09T21:55:44.401 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 138/192 2024-06-09T21:55:45.350 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libtool-2.4.6-45.el9.x86_64 139/192 2024-06-09T21:55:45.382 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 140/192 2024-06-09T21:55:45.406 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 140/192 2024-06-09T21:55:45.406 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:45.407 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-09T21:55:45.407 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T21:55:45.407 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-09T21:55:45.407 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:45.457 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 141/192 2024-06-09T21:55:45.480 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 141/192 2024-06-09T21:55:45.480 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:45.481 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-09T21:55:45.481 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:45.533 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/192 2024-06-09T21:55:45.556 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 142/192 2024-06-09T21:55:45.557 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:45.557 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-06-09T21:55:45.557 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T21:55:45.557 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-06-09T21:55:45.557 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:45.875 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 143/192 2024-06-09T21:55:45.899 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 143/192 2024-06-09T21:55:45.899 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-09T21:55:45.899 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-09T21:55:45.899 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T21:55:45.899 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-09T21:55:45.900 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:55:48.563 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 144/192 2024-06-09T21:55:48.605 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 145/192 2024-06-09T21:55:48.654 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 146/192 2024-06-09T21:55:48.686 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 147/192 2024-06-09T21:55:48.713 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/192 2024-06-09T21:55:48.903 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 149/192 2024-06-09T21:55:48.951 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/192 2024-06-09T21:55:48.984 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 151/192 2024-06-09T21:55:49.032 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/192 2024-06-09T21:55:49.062 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 153/192 2024-06-09T21:55:49.108 INFO:teuthology.orchestra.run.smithi148.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 154/192 2024-06-09T21:55:49.714 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/192 2024-06-09T21:55:49.748 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/192 2024-06-09T21:55:49.767 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/192 2024-06-09T21:55:50.695 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 144/192 2024-06-09T21:55:50.738 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 145/192 2024-06-09T21:55:50.778 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 146/192 2024-06-09T21:55:50.810 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 147/192 2024-06-09T21:55:50.837 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : elfutils-libelf-devel-0.191-4.el9.x86_64 148/192 2024-06-09T21:55:51.049 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 149/192 2024-06-09T21:55:51.091 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 150/192 2024-06-09T21:55:51.124 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 151/192 2024-06-09T21:55:51.164 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : elfutils-debuginfod-client-0.191-4.el9.x86_64 152/192 2024-06-09T21:55:51.194 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-team-1:1.47.91-1.el9.x86_64 153/192 2024-06-09T21:55:51.239 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : NetworkManager-tui-1:1.47.91-1.el9.x86_64 154/192 2024-06-09T21:55:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/192 2024-06-09T21:55:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 155/192 2024-06-09T21:55:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 155/192 2024-06-09T21:55:56.893 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 156/192 2024-06-09T21:55:59.417 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 156/192 2024-06-09T21:56:40.025 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9. 157/192 2024-06-09T21:56:40.808 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el 158/192 2024-06-09T21:56:40.985 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9. 159/192 2024-06-09T21:56:45.132 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9. 157/192 2024-06-09T21:56:45.818 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el 158/192 2024-06-09T21:56:45.895 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 160/192 2024-06-09T21:56:45.998 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9. 159/192 2024-06-09T21:56:48.179 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 161/192 2024-06-09T21:56:51.210 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 160/192 2024-06-09T21:56:53.420 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 161/192 2024-06-09T21:56:57.871 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 162/192 2024-06-09T21:57:03.692 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 162/192 2024-06-09T21:57:25.904 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 163/192 2024-06-09T21:57:33.578 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 163/192 2024-06-09T21:57:44.154 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 164/192 2024-06-09T21:57:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 164/192 2024-06-09T21:58:53.401 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.e 165/192 2024-06-09T21:58:54.229 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9. 166/192 2024-06-09T21:58:54.532 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9 167/192 2024-06-09T21:58:59.536 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2946.g3d89accf 168/192 2024-06-09T21:59:05.176 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9. 169/192 2024-06-09T21:59:05.259 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 170/192 2024-06-09T21:59:05.372 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf 171/192 2024-06-09T21:59:05.518 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2946.g3d89accf. 172/192 2024-06-09T21:59:05.575 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el 173/192 2024-06-09T21:59:05.656 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el 174/192 2024-06-09T21:59:06.247 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.e 165/192 2024-06-09T21:59:07.099 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9. 166/192 2024-06-09T21:59:07.188 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9 167/192 2024-06-09T21:59:11.216 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 175/192 2024-06-09T21:59:11.616 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9 176/192 2024-06-09T21:59:12.260 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-debuginfo-2:19.0.0-2946.g3d89accf 168/192 2024-06-09T21:59:12.482 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 177/192 2024-06-09T21:59:12.571 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 178/192 2024-06-09T21:59:12.618 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 179/192 2024-06-09T21:59:12.704 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : flex-2.6.4-9.el9.x86_64 180/192 2024-06-09T21:59:12.734 INFO:teuthology.orchestra.run.smithi148.stdout: Installing : bison-3.7.4-5.el9.x86_64 181/192 2024-06-09T21:59:12.759 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 182/192 2024-06-09T21:59:12.785 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 183/192 2024-06-09T21:59:12.812 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 184/192 2024-06-09T21:59:12.812 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 185/192 2024-06-09T21:59:12.896 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 185/192 2024-06-09T21:59:12.930 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 186/192 2024-06-09T21:59:12.930 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 187/192 2024-06-09T21:59:13.013 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 187/192 2024-06-09T21:59:13.038 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 188/192 2024-06-09T21:59:13.050 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 189/192 2024-06-09T21:59:13.050 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 189/192 2024-06-09T21:59:13.234 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 189/192 2024-06-09T21:59:13.234 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 190/192 2024-06-09T21:59:13.326 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 190/192 2024-06-09T21:59:13.351 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 191/192 2024-06-09T21:59:13.417 INFO:teuthology.orchestra.run.smithi148.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 192/192 2024-06-09T21:59:18.243 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9. 169/192 2024-06-09T21:59:18.329 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 170/192 2024-06-09T21:59:18.453 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf 171/192 2024-06-09T21:59:18.591 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-debuginfo-2:19.0.0-2946.g3d89accf. 172/192 2024-06-09T21:59:18.653 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el 173/192 2024-06-09T21:59:18.735 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el 174/192 2024-06-09T21:59:22.774 INFO:teuthology.orchestra.run.smithi148.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 192/192 2024-06-09T21:59:22.774 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/192 2024-06-09T21:59:22.774 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2/192 2024-06-09T21:59:22.774 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9. 3/192 2024-06-09T21:59:22.774 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 4/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el 5/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 6/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 7/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9. 8/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 9/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 10/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 11/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 12/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 13/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 14/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 15/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 16/192 2024-06-09T21:59:22.775 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 17/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 18/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 19/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.e 20/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 21/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 22/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9. 23/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 24/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 25/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 26/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 27/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9 28/192 2024-06-09T21:59:22.776 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 29/192 2024-06-09T21:59:22.790 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 30/192 2024-06-09T21:59:22.790 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2946.g3d89accf 31/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9. 32/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 33/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 34/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 35/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 36/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 37/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 38/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf 39/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 40/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2946.g3d89accf. 41/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 42/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el 43/192 2024-06-09T21:59:22.791 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 44/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el 45/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 46/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 47/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 48/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9 49/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 50/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 51/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 52/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 53/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 54/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 55/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 56/192 2024-06-09T21:59:22.792 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 57/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 58/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 59/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 60/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 61/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 62/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 63/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 64/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 65/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 66/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-ply-3.11-14.el9.noarch 67/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 68/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 69/192 2024-06-09T21:59:22.793 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 70/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : automake-1.16.2-8.el9.noarch 71/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : bison-3.7.4-5.el9.x86_64 72/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 73/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 74/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flex-2.6.4-9.el9.x86_64 75/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 76/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 77/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 78/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 79/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 80/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 81/192 2024-06-09T21:59:22.794 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 82/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 83/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 84/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 85/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 86/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lua-5.4.4-4.el9.x86_64 87/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 88/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 89/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 90/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-13.14-1.el9.x86_64 91/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 92/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 93/192 2024-06-09T21:59:22.795 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 94/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 95/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 96/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 97/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 98/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 99/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 100/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 101/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 102/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 103/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 104/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 105/192 2024-06-09T21:59:22.796 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 106/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 107/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 108/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 109/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 110/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 111/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 112/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 113/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 114/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 115/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 116/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 117/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 118/192 2024-06-09T21:59:22.797 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 119/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 120/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 121/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 122/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 123/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 124/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 125/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 126/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 127/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 128/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 129/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 130/192 2024-06-09T21:59:22.798 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 131/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 132/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 133/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 134/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 135/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 136/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 137/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 138/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 139/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 140/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 141/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 142/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 143/192 2024-06-09T21:59:22.799 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 144/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 145/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 146/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 147/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 148/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 149/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 150/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 151/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 152/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 153/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 154/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 155/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 156/192 2024-06-09T21:59:22.800 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 157/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 158/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 159/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 160/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 161/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 162/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 163/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 164/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 165/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 166/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 167/192 2024-06-09T21:59:22.801 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 168/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : re2-1:20211101-3.el9.x86_64 169/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 170/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 171/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 172/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 173/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 174/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 175/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 176/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 177/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 178/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 179/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 180/192 2024-06-09T21:59:22.802 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 181/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 182/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 183/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 184/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 185/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 186/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 187/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 188/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 189/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 190/192 2024-06-09T21:59:22.803 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 191/192 2024-06-09T21:59:24.618 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 175/192 2024-06-09T21:59:24.788 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9 176/192 2024-06-09T21:59:25.707 INFO:teuthology.orchestra.run.smithi148.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 192/192 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout:Upgraded: 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-09T21:59:25.708 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout:Installed: 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: automake-1.16.2-8.el9.noarch 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: bison-3.7.4-5.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.709 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.710 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-09T21:59:25.711 INFO:teuthology.orchestra.run.smithi148.stdout: flex-2.6.4-9.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.712 INFO:teuthology.orchestra.run.smithi148.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-09T21:59:25.713 INFO:teuthology.orchestra.run.smithi148.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: lua-5.4.4-4.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-13.14-1.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-09T21:59:25.714 INFO:teuthology.orchestra.run.smithi148.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-09T21:59:25.715 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-09T21:59:25.716 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-ply-3.11-14.el9.noarch 2024-06-09T21:59:25.717 INFO:teuthology.orchestra.run.smithi148.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-09T21:59:25.718 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-09T21:59:25.719 INFO:teuthology.orchestra.run.smithi148.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: re2-1:20211101-3.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-09T21:59:25.720 INFO:teuthology.orchestra.run.smithi148.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-09T21:59:25.721 INFO:teuthology.orchestra.run.smithi148.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-09T21:59:25.721 INFO:teuthology.orchestra.run.smithi148.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-09T21:59:25.721 INFO:teuthology.orchestra.run.smithi148.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-09T21:59:25.721 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 177/192 2024-06-09T21:59:25.721 INFO:teuthology.orchestra.run.smithi148.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-09T21:59:25.721 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:25.722 INFO:teuthology.orchestra.run.smithi148.stdout:Complete! 2024-06-09T21:59:25.813 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 178/192 2024-06-09T21:59:25.878 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 179/192 2024-06-09T21:59:25.977 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flex-2.6.4-9.el9.x86_64 180/192 2024-06-09T21:59:26.010 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : bison-3.7.4-5.el9.x86_64 181/192 2024-06-09T21:59:26.034 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 182/192 2024-06-09T21:59:26.059 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-debuginfod-client-0.190-2.el9.x86_64 183/192 2024-06-09T21:59:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-libs-0.190-2.el9.x86_64 184/192 2024-06-09T21:59:26.086 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 185/192 2024-06-09T21:59:26.170 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 185/192 2024-06-09T21:59:26.214 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 186/192 2024-06-09T21:59:26.214 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 187/192 2024-06-09T21:59:26.314 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 187/192 2024-06-09T21:59:26.347 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 188/192 2024-06-09T21:59:26.359 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 189/192 2024-06-09T21:59:26.359 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 189/192 2024-06-09T21:59:26.366 DEBUG:teuthology.parallel:result is None 2024-06-09T21:59:26.549 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 189/192 2024-06-09T21:59:26.549 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 190/192 2024-06-09T21:59:26.625 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 190/192 2024-06-09T21:59:26.649 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 191/192 2024-06-09T21:59:26.718 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : elfutils-libelf-0.190-2.el9.x86_64 192/192 2024-06-09T21:59:36.065 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: elfutils-libelf-0.190-2.el9.x86_64 192/192 2024-06-09T21:59:36.065 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 1/192 2024-06-09T21:59:36.065 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2/192 2024-06-09T21:59:36.065 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9. 3/192 2024-06-09T21:59:36.065 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 4/192 2024-06-09T21:59:36.065 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el 5/192 2024-06-09T21:59:36.065 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_6 6/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 7/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9. 8/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2946.g3d89a 9/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-2 10/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 11/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 12/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 13/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 14/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 15/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 16/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 17/192 2024-06-09T21:59:36.066 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 18/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 19/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.e 20/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 21/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 22/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9. 23/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 24/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 25/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_ 26/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 27/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9 28/192 2024-06-09T21:59:36.067 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 29/192 2024-06-09T21:59:36.082 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_6 30/192 2024-06-09T21:59:36.082 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-2946.g3d89accf 31/192 2024-06-09T21:59:36.082 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9. 32/192 2024-06-09T21:59:36.082 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86 33/192 2024-06-09T21:59:36.082 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 34/192 2024-06-09T21:59:36.082 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 35/192 2024-06-09T21:59:36.082 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el 36/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.0.0-2946.g3d89accf.el9. 37/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_6 38/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf 39/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 40/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-2946.g3d89accf. 41/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 42/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el 43/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 44/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el 45/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 46/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x 47/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 48/192 2024-06-09T21:59:36.083 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9 49/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 50/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x8 51/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf. 52/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noa 53/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.n 54/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d8 55/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el 56/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 57/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.e 58/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 59/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 60/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 61/192 2024-06-09T21:59:36.084 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 62/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 63/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 64/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 65/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 66/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 67/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 68/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 69/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 70/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : automake-1.16.2-8.el9.noarch 71/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : bison-3.7.4-5.el9.x86_64 72/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 73/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-devel-0.191-4.el9.x86_64 74/192 2024-06-09T21:59:36.085 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flex-2.6.4-9.el9.x86_64 75/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 76/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 77/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 78/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 79/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 80/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 81/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 82/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 83/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 84/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 85/192 2024-06-09T21:59:36.086 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 86/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 87/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 88/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 89/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 90/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-13.14-1.el9.x86_64 91/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 92/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 93/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 94/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 95/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 96/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 97/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-colorama-0.4.6-3.el9.noarch 98/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 99/192 2024-06-09T21:59:36.087 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 100/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 101/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 102/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 103/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 104/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 105/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 106/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 107/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 108/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 109/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 110/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 111/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 112/192 2024-06-09T21:59:36.088 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 113/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 114/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 115/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 116/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 117/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 118/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 119/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 120/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 121/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 122/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 123/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 124/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 125/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 126/192 2024-06-09T21:59:36.089 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 127/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 128/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 129/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 130/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 131/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 132/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 133/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 134/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 135/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 136/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 137/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 138/192 2024-06-09T21:59:36.090 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 139/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 140/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 141/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 142/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 143/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 144/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 145/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 146/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 147/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 148/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 149/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 150/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 151/192 2024-06-09T21:59:36.091 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 152/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 153/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 154/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 155/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 156/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 157/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 158/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 159/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 160/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 161/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 162/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 163/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 164/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 165/192 2024-06-09T21:59:36.092 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 166/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 167/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 168/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-3.el9.x86_64 169/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 170/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 171/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 172/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 173/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 174/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-1:1.47.91-1.el9.x86_64 175/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 176/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-libnm-1:1.47.91-1.el9.x86_64 177/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 178/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-team-1:1.47.91-1.el9.x86_64 179/192 2024-06-09T21:59:36.093 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 180/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-tui-1:1.47.91-1.el9.x86_64 181/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 182/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-debuginfod-client-0.191-4.el9.x86_64 183/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 184/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-0.191-4.el9.x86_64 185/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 186/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libs-0.191-4.el9.x86_64 187/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 188/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 189/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 190/192 2024-06-09T21:59:36.094 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 191/192 2024-06-09T21:59:38.962 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 192/192 2024-06-09T21:59:38.962 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-1:1.47.91-1.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-libnm-1:1.47.91-1.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-team-1:1.47.91-1.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: NetworkManager-tui-1:1.47.91-1.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-debuginfod-client-0.191-4.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-0.191-4.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libs-0.191-4.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-06-09T21:59:38.963 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: automake-1.16.2-8.el9.noarch 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: bison-3.7.4-5.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.964 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.965 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.966 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top-2:19.0.0-2946.g3d89accf.el9.noarch 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel-0.191-4.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: flex-2.6.4-9.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: fuse-2.9.9-15.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-09T21:59:38.967 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.968 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: libtool-2.4.6-45.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-5.el9.noarch 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-09T21:59:38.969 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-13.14-1.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server-13.14-1.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-09T21:59:38.970 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama-0.4.6-3.el9.noarch 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-09T21:59:38.971 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-09T21:59:38.972 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-09T21:59:38.973 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-09T21:59:38.974 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-09T21:59:38.975 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-24.02.0-1.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-debuginfo-2:19.0.0-2946.g3d89accf.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-3.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-09T21:59:38.976 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-09T21:59:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: uuid-1.6.2-55.el9.x86_64 2024-06-09T21:59:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-06-09T21:59:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-09T21:59:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-06-09T21:59:38.977 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:38.977 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-06-09T21:59:39.737 DEBUG:teuthology.parallel:result is None 2024-06-09T21:59:39.738 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T21:59:39.891 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-09T21:59:39.919 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-2946.g3d89accf.el9 2024-06-09T21:59:39.920 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-09T21:59:39.920 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-09T21:59:39.922 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3d89accfe3201286f000e065a1faea2a88c7a6ba 2024-06-09T21:59:40.060 DEBUG:teuthology.orchestra.run.smithi148:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-09T21:59:40.087 INFO:teuthology.orchestra.run.smithi148.stdout:19.0.0-2946.g3d89accf.el9 2024-06-09T21:59:40.087 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2946.g3d89accf.el9 2024-06-09T21:59:40.088 INFO:teuthology.task.install:The correct ceph version 19.0.0-2946.g3d89accf is installed. 2024-06-09T21:59:40.089 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-09T21:59:40.089 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:40.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-09T21:59:40.123 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:59:40.124 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-09T21:59:40.159 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-09T21:59:40.159 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:40.159 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-06-09T21:59:40.192 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-09T21:59:40.260 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:59:40.260 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/daemon-helper 2024-06-09T21:59:40.288 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-09T21:59:40.355 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-09T21:59:40.356 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:40.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-09T21:59:40.384 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-09T21:59:40.453 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:59:40.454 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-09T21:59:40.481 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-09T21:59:40.547 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-09T21:59:40.547 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:40.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-06-09T21:59:40.576 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-09T21:59:40.643 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:59:40.643 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/usr/bin/stdin-killer 2024-06-09T21:59:40.669 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-09T21:59:40.736 INFO:teuthology.run_tasks:Running task ceph... 2024-06-09T21:59:40.813 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-06-09T21:59:40.813 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-06-09T21:59:40.816 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 777 /var/log/ceph 2024-06-09T21:59:40.840 INFO:tasks.ceph:Disabling ceph logrotate... 2024-06-09T21:59:40.841 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-09T21:59:40.882 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-09T21:59:40.907 INFO:tasks.ceph:Creating extra log directories... 2024-06-09T21:59:40.907 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-09T21:59:40.949 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-09T21:59:40.980 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-06-09T21:59:40.981 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-06-09T21:59:40.981 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-06-09T21:59:40.981 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:40.981 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-06-09T21:59:41.030 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-06-09T21:59:41.146 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-06-09T21:59:41.172 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:59:41.172 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-06-09T21:59:41.188 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-06-09T21:59:41.298 DEBUG:teuthology.orchestra.run.smithi148:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-06-09T21:59:41.327 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-06-09T21:59:41.327 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-06-09T21:59:41.328 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204', 'branch': 'squid', 'description': 'fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7747204', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'ceph-fuse': {'client.0': {'valgrind': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': '7b7a5d375cf4e4afab96b9d4d229b699ae8304bb'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 178, 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'sleep_before_teardown': 0, 'subset': '130/512', 'suite': 'fs', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7b7a5d375cf4e4afab96b9d4d229b699ae8304bb', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCB91KijbRW8o/02HQs4ZPWIzY7nw/kn9ggUOIz4B+uLmnaVz4tJHt/SmsTFqAn1hQzzQys5R1jzk5qyn3zGbVM=', 'smithi148.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMF/mae4dPj95xN4d/T7HfQkyRz/7xjOcRo9mLSElNtAnQmqDkXpqzhCHMBf1NSH+KDwdwBWx3o/bZ0/T744NgQ='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'debuginfo': True, 'flavor': 'default', 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow request', 'SLOW_OPS', 'MON_DOWN'], 'sha1': '3d89accfe3201286f000e065a1faea2a88c7a6ba', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes']}, 'watchdog': {'daemon_restart': 'normal'}, 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'workunit': {'clients': {'all': ['suites/dbench.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '861a8dcf7aa816a26e13f039336f7ed0a9aec0fa', 'timestamp': '2024-06-09_21:24:02', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-06-09T21:59:41.328 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-09T21:59:41.330 DEBUG:teuthology.orchestra.run.smithi148:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-09T21:59:41.378 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-06-09T21:59:41.386 DEBUG:teuthology.orchestra.run.smithi148:> sudo install -d -m0777 -- /var/run/ceph 2024-06-09T21:59:41.442 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:41.442 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-06-09T21:59:41.467 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-09T21:59:41.467 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-06-09T21:59:41.523 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-09T21:59:41.523 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T21:59:41.523 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-06-09T21:59:41.523 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T21:59:41.523 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T21:59:41.523 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-06-09 21:59:27.085044264 +0000 2024-06-09T21:59:41.524 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-06-09 21:50:34.490635194 +0000 2024-06-09T21:59:41.524 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-06-09 21:50:34.490635194 +0000 2024-06-09T21:59:41.524 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-06-09 21:50:34.490635194 +0000 2024-06-09T21:59:41.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-09T21:59:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-06-09T21:59:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-06-09T21:59:41.587 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000101957 s, 5.0 MB/s 2024-06-09T21:59:41.588 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-09T21:59:41.646 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-06-09 21:59:27.085044264 +0000 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-06-09 21:50:34.818628859 +0000 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-06-09 21:50:34.818628859 +0000 2024-06-09T21:59:41.702 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-06-09 21:50:34.818628859 +0000 2024-06-09T21:59:41.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-09T21:59:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-06-09T21:59:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-06-09T21:59:41.766 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000100776 s, 5.1 MB/s 2024-06-09T21:59:41.768 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-09T21:59:41.824 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-06-09T21:59:41.880 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-09T21:59:41.880 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T21:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-06-09T21:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T21:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T21:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-06-09 21:59:27.086044244 +0000 2024-06-09T21:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-06-09 21:50:35.153622390 +0000 2024-06-09T21:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-06-09 21:50:35.153622390 +0000 2024-06-09T21:59:41.881 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-06-09 21:50:35.153622390 +0000 2024-06-09T21:59:41.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-09T21:59:41.945 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-06-09T21:59:41.945 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-06-09T21:59:41.945 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000100214 s, 5.1 MB/s 2024-06-09T21:59:41.946 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-09T21:59:42.003 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-06-09T21:59:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-09T21:59:42.059 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T21:59:42.059 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-06-09T21:59:42.059 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T21:59:42.059 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T21:59:42.059 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-06-09 21:59:27.086044244 +0000 2024-06-09T21:59:42.059 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-06-09 21:50:35.481616055 +0000 2024-06-09T21:59:42.059 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-06-09 21:50:35.481616055 +0000 2024-06-09T21:59:42.060 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-06-09 21:50:35.481616055 +0000 2024-06-09T21:59:42.060 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-09T21:59:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-06-09T21:59:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-06-09T21:59:42.122 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000108661 s, 4.7 MB/s 2024-06-09T21:59:42.124 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-09T21:59:42.180 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:42.180 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:59:42.180 DEBUG:teuthology.orchestra.run.smithi148:> dd if=/scratch_devs of=/dev/stdout 2024-06-09T21:59:42.198 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-09T21:59:42.198 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_1 2024-06-09T21:59:42.254 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-09T21:59:42.255 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T21:59:42.255 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 670 Links: 1 2024-06-09T21:59:42.255 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T21:59:42.255 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T21:59:42.255 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-06-09 21:59:13.772859982 +0000 2024-06-09T21:59:42.255 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-06-09 21:50:32.629233464 +0000 2024-06-09T21:59:42.255 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-06-09 21:50:32.629233464 +0000 2024-06-09T21:59:42.255 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-06-09 21:50:32.629233464 +0000 2024-06-09T21:59:42.255 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-09T21:59:42.320 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-06-09T21:59:42.321 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-06-09T21:59:42.321 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000194244 s, 2.6 MB/s 2024-06-09T21:59:42.322 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-09T21:59:42.380 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_2 2024-06-09T21:59:42.437 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-09T21:59:42.437 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T21:59:42.437 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-06-09T21:59:42.437 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T21:59:42.437 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T21:59:42.437 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-06-09 21:59:13.773859959 +0000 2024-06-09T21:59:42.438 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-06-09 21:50:32.964225498 +0000 2024-06-09T21:59:42.438 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-06-09 21:50:32.964225498 +0000 2024-06-09T21:59:42.438 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-06-09 21:50:32.964225498 +0000 2024-06-09T21:59:42.438 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-09T21:59:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-06-09T21:59:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-06-09T21:59:42.501 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000237986 s, 2.2 MB/s 2024-06-09T21:59:42.502 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-09T21:59:42.559 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_3 2024-06-09T21:59:42.615 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-09T21:59:42.615 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T21:59:42.615 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 708 Links: 1 2024-06-09T21:59:42.615 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T21:59:42.615 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T21:59:42.615 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-06-09 21:59:13.773859959 +0000 2024-06-09T21:59:42.615 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-06-09 21:50:33.315217151 +0000 2024-06-09T21:59:42.616 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-06-09 21:50:33.315217151 +0000 2024-06-09T21:59:42.616 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-06-09 21:50:33.315217151 +0000 2024-06-09T21:59:42.616 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-09T21:59:42.678 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-06-09T21:59:42.678 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-06-09T21:59:42.678 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000206231 s, 2.5 MB/s 2024-06-09T21:59:42.679 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-09T21:59:42.736 DEBUG:teuthology.orchestra.run.smithi148:> stat /dev/vg_nvme/lv_4 2024-06-09T21:59:42.791 INFO:teuthology.orchestra.run.smithi148.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-09T21:59:42.791 INFO:teuthology.orchestra.run.smithi148.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-09T21:59:42.792 INFO:teuthology.orchestra.run.smithi148.stdout:Device: 5h/5d Inode: 727 Links: 1 2024-06-09T21:59:42.792 INFO:teuthology.orchestra.run.smithi148.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-09T21:59:42.792 INFO:teuthology.orchestra.run.smithi148.stdout:Context: system_u:object_r:device_t:s0 2024-06-09T21:59:42.792 INFO:teuthology.orchestra.run.smithi148.stdout:Access: 2024-06-09 21:59:13.774859936 +0000 2024-06-09T21:59:42.792 INFO:teuthology.orchestra.run.smithi148.stdout:Modify: 2024-06-09 21:50:33.640209422 +0000 2024-06-09T21:59:42.792 INFO:teuthology.orchestra.run.smithi148.stdout:Change: 2024-06-09 21:50:33.640209422 +0000 2024-06-09T21:59:42.792 INFO:teuthology.orchestra.run.smithi148.stdout: Birth: 2024-06-09 21:50:33.640209422 +0000 2024-06-09T21:59:42.792 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-09T21:59:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records in 2024-06-09T21:59:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:1+0 records out 2024-06-09T21:59:42.855 INFO:teuthology.orchestra.run.smithi148.stderr:512 bytes copied, 0.000180215 s, 2.8 MB/s 2024-06-09T21:59:42.856 DEBUG:teuthology.orchestra.run.smithi148:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-09T21:59:42.913 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:42.913 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-09T21:59:42.913 INFO:tasks.ceph:Generating config... 2024-06-09T21:59:42.914 INFO:tasks.ceph:[client] client mount timeout = 600 2024-06-09T21:59:42.914 INFO:tasks.ceph:[client] debug client = 20 2024-06-09T21:59:42.914 INFO:tasks.ceph:[client] debug ms = 1 2024-06-09T21:59:42.914 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-06-09T21:59:42.915 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-06-09T21:59:42.915 INFO:tasks.ceph:[global] mds beacon grace = 60 2024-06-09T21:59:42.915 INFO:tasks.ceph:[global] mds heartbeat grace = 60 2024-06-09T21:59:42.915 INFO:tasks.ceph:[global] osd heartbeat grace = 60 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] debug mds = 20 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] debug ms = 1 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] mds debug frag = True 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] mds valgrind exit = True 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-06-09T21:59:42.915 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mgr] debug client = 20 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mgr] debug ms = 1 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mon] debug mon = 20 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mon] debug ms = 1 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mon] debug paxos = 20 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-06-09T21:59:42.916 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-06-09T21:59:42.916 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-06-09T21:59:42.916 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-06-09T21:59:42.916 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-06-09T21:59:42.916 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-06-09T21:59:42.916 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] debug ms = 1 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] debug osd = 20 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-06-09T21:59:42.917 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-06-09T21:59:42.917 INFO:tasks.ceph:Setting up mon.a... 2024-06-09T21:59:42.918 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-06-09T21:59:42.975 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.keyring 2024-06-09T21:59:42.978 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-06-09T21:59:43.026 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-09T21:59:43.053 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.32'), ('mon.b', '172.21.15.148'), ('mon.c', '[v2:172.21.15.148:3301,v1:172.21.15.148:6790]')] 2024-06-09T21:59:43.053 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.32,172.21.15.148,[v2:172.21.15.148:3301,v1:172.21.15.148:6790]', 'mds beacon grace': 60, 'mds heartbeat grace': 60, 'osd heartbeat grace': 60}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120, 'mon osd crush smoke test': False}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds valgrind exit': True, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-06-09T21:59:43.054 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:43.054 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-06-09T21:59:43.109 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.32 --add b 172.21.15.148 --addv c '[v2:172.21.15.148:3301,v1:172.21.15.148:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-06-09T21:59:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-06-09T21:59:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: generated fsid 08d1d92e-3533-4051-bd41-97c73d8b6b19 2024-06-09T21:59:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-06-09T21:59:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 0 2024-06-09T21:59:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:fsid 08d1d92e-3533-4051-bd41-97c73d8b6b19 2024-06-09T21:59:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:last_changed 2024-06-09T21:59:43.214134+0000 2024-06-09T21:59:43.197 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-06-09T21:59:43.214134+0000 2024-06-09T21:59:43.198 INFO:teuthology.orchestra.run.smithi032.stdout:min_mon_release 17 (quincy) 2024-06-09T21:59:43.198 INFO:teuthology.orchestra.run.smithi032.stdout:election_strategy: 1 2024-06-09T21:59:43.198 INFO:teuthology.orchestra.run.smithi032.stdout:0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-06-09T21:59:43.198 INFO:teuthology.orchestra.run.smithi032.stdout:1: [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] mon.b 2024-06-09T21:59:43.198 INFO:teuthology.orchestra.run.smithi032.stdout:2: [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] mon.c 2024-06-09T21:59:43.198 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-06-09T21:59:43.199 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-06-09T21:59:43.215 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 08d1d92e-3533-4051-bd41-97c73d8b6b19... 2024-06-09T21:59:43.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-09T21:59:43.257 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-09T21:59:43.299 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-06-09T21:59:43.299 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-06-09T21:59:43.299 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-09T21:59:43.299 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-06-09T21:59:43.299 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.299 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-06-09T21:59:43.299 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.300 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-06-09T21:59:43.301 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-06-09T21:59:43.302 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "172.21.15.32,172.21.15.148,[v2:172.21.15.148:3301,v1:172.21.15.148:6790]" 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: mds beacon grace = 60 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: mds heartbeat grace = 60 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: osd heartbeat grace = 60 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = 08d1d92e-3533-4051-bd41-97c73d8b6b19 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-06-09T21:59:43.303 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-06-09T21:59:43.304 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-06-09T21:59:43.305 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore compression mode = aggressive 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 20 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 20 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 10 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-06-09T21:59:43.306 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: osd fast shutdown = False 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-06-09T21:59:43.307 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-06-09T21:59:43.308 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-06-09T21:59:43.309 INFO:teuthology.orchestra.run.smithi032.stdout: mon down mkfs grace = 300 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: mon op complaint time = 120 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd crush smoke test = False 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: client mount timeout = 600 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: debug client = 20 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-06-09T21:59:43.310 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.e] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.f] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout:[mds] 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds = 20 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout: debug mds balancer = 20 2024-06-09T21:59:43.311 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-06-09T21:59:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug frag = True 2024-06-09T21:59:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: mds debug scatterstat = True 2024-06-09T21:59:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: mds op complaint time = 180 2024-06-09T21:59:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: mds valgrind exit = True 2024-06-09T21:59:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: mds verify scatter = True 2024-06-09T21:59:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: osd op complaint time = 180 2024-06-09T21:59:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: rados mon op timeout = 900 2024-06-09T21:59:43.312 INFO:teuthology.orchestra.run.smithi032.stdout: rados osd op timeout = 900 2024-06-09T21:59:43.313 INFO:teuthology.orchestra.run.smithi148.stdout:[global] 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: chdir = "" 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: auth supported = cephx 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: filestore xattr use omap = true 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: mon clock drift allowed = 1.000 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: osd crush chooseleaf type = 0 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: auth debug = true 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.314 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on old message = true 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: ms die on bug = true 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: mon pg warn max object skew = 0 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: # disable pg_autoscaler by default for new pools 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default size = 2 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow primary affinity = true 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd allow pg remap = true 2024-06-09T21:59:43.315 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on legacy crush tunables = false 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on crush straw calc version zero = false 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on no sortbitwise = false 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on osd down out interval zero = false 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: mon warn on too few osds = false 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: mon_allow_pool_size_one = true 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: osd default data pool replay window = 5 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: mon allow pool delete = true 2024-06-09T21:59:43.316 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: mon cluster log file level = debug 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: debug asserts on shutdown = true 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: mon health detail to clog = false 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: mon host = "172.21.15.32,172.21.15.148,[v2:172.21.15.148:3301,v1:172.21.15.148:6790]" 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: mds beacon grace = 60 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: mds heartbeat grace = 60 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: osd heartbeat grace = 60 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: fsid = 08d1d92e-3533-4051-bd41-97c73d8b6b19 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout:[osd] 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: osd journal size = 100 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.317 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub load threshold = 5.0 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd scrub max interval = 600 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd mclock profile = high_recovery_ops 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd recover clone overlap = true 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd recovery max chunk = 1048576 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug shutdown = true 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug op order = true 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug verify stray on activate = true 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug trim objects = true 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.318 INFO:teuthology.orchestra.run.smithi148.stdout: osd open classes on start = true 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug pg log writeout = true 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: osd deep scrub update digest min age = 30 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: osd map max advance = 10 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: journal zero on create = true 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: filestore ondisk finisher threads = 3 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: filestore apply finisher threads = 3 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: bdev debug aio = true 2024-06-09T21:59:43.319 INFO:teuthology.orchestra.run.smithi148.stdout: osd debug misdirected ops = true 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore block size = 96636764160 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore compression mode = aggressive 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: bluestore fsck on mount = True 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluefs = 20 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: debug bluestore = 20 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: debug osd = 20 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: debug rocksdb = 10 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd backfillfull_ratio = 0.85 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd full ratio = 0.9 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd nearfull ratio = 0.8 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: osd failsafe full ratio = 0.95 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: osd fast shutdown = False 2024-06-09T21:59:43.320 INFO:teuthology.orchestra.run.smithi148.stdout: osd objectstore = bluestore 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: osd op complaint time = 180 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr] 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: debug mgr = 20 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: mgr/telemetry/nag = false 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: debug client = 20 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.321 INFO:teuthology.orchestra.run.smithi148.stdout:[mon] 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: debug mon = 20 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: debug paxos = 20 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: debug auth = 20 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: mon data avail warn = 5 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: mon mgr mkfs grace = 240 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min pgs per osd = 4 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd reporter subtree level = osd 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd prime pg temp = true 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: mon reweight min bytes per osd = 10 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.322 INFO:teuthology.orchestra.run.smithi148.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: auth mon ticket ttl = 660 # 11m 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: auth service ticket ttl = 240 # 4m 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: # don't complain about insecure global_id in the test suite 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: # 1m isn't quite enough 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: mon_down_mkfs_grace = 2m 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: mon_warn_on_filestore_osds = false 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: mon down mkfs grace = 300 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: mon op complaint time = 120 2024-06-09T21:59:43.323 INFO:teuthology.orchestra.run.smithi148.stdout: mon osd crush smoke test = False 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout:[client] 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: rgw cache enabled = true 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable ops log = true 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: rgw enable usage log = true 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: client mount timeout = 600 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: debug client = 20 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: rados mon op timeout = 900 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout: rados osd op timeout = 900 2024-06-09T21:59:43.324 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.a] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.b] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mon.c] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.a] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.c] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.e] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.b] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.d] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.f] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout:[mds] 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout: debug mds = 20 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout: debug mds balancer = 20 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout: debug ms = 1 2024-06-09T21:59:43.325 INFO:teuthology.orchestra.run.smithi148.stdout: mds debug frag = True 2024-06-09T21:59:43.326 INFO:teuthology.orchestra.run.smithi148.stdout: mds debug scatterstat = True 2024-06-09T21:59:43.326 INFO:teuthology.orchestra.run.smithi148.stdout: mds op complaint time = 180 2024-06-09T21:59:43.326 INFO:teuthology.orchestra.run.smithi148.stdout: mds valgrind exit = True 2024-06-09T21:59:43.326 INFO:teuthology.orchestra.run.smithi148.stdout: mds verify scatter = True 2024-06-09T21:59:43.326 INFO:teuthology.orchestra.run.smithi148.stdout: osd op complaint time = 180 2024-06-09T21:59:43.326 INFO:teuthology.orchestra.run.smithi148.stdout: rados mon op timeout = 900 2024-06-09T21:59:43.326 INFO:teuthology.orchestra.run.smithi148.stdout: rados osd op timeout = 900 2024-06-09T21:59:43.326 INFO:tasks.ceph:Creating admin key on mon.a... 2024-06-09T21:59:43.326 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-06-09T21:59:43.409 INFO:tasks.ceph:Copying monmap to all nodes... 2024-06-09T21:59:43.409 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:43.409 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-06-09T21:59:43.425 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:43.425 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-06-09T21:59:43.480 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T21:59:43.480 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:43.480 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-09T21:59:43.481 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-09T21:59:43.556 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T21:59:43.556 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-09T21:59:43.613 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T21:59:43.613 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:59:43.613 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-09T21:59:43.613 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-09T21:59:43.648 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T21:59:43.648 DEBUG:teuthology.orchestra.run.smithi148:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-09T21:59:43.704 INFO:tasks.ceph:Setting up mon nodes... 2024-06-09T21:59:43.705 INFO:tasks.ceph:Setting up mgr nodes... 2024-06-09T21:59:43.705 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-06-09T21:59:43.757 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-06-09T21:59:43.760 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-06-09T21:59:43.827 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-06-09T21:59:43.830 INFO:tasks.ceph:Setting up mds nodes... 2024-06-09T21:59:43.830 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-06-09T21:59:43.883 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-06-09T21:59:43.885 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-06-09T21:59:43.911 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-06-09T21:59:44.003 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-06-09T21:59:44.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-06-09T21:59:44.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-06-09T21:59:44.122 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-06-09T21:59:44.125 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-06-09T21:59:44.151 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-06-09T21:59:44.202 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-06-09T21:59:44.205 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-06-09T21:59:44.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-06-09T21:59:44.322 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-06-09T21:59:44.325 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-06-09T21:59:44.349 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-06-09T21:59:44.443 INFO:teuthology.orchestra.run.smithi148.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-06-09T21:59:44.446 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-06-09T21:59:44.471 INFO:tasks.ceph_client:Setting up client nodes... 2024-06-09T21:59:44.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-09T21:59:44.515 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-06-09T21:59:44.529 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-06-09T21:59:44.529 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi148.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-09T21:59:44.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-06-09T21:59:44.555 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:44.555 INFO:tasks.ceph:role: osd.0 2024-06-09T21:59:44.555 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T21:59:44.555 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-09T21:59:44.620 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-09T21:59:44.621 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-09T21:59:44.661 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-06-09T21:59:44.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-06-09T21:59:44.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-06-09T21:59:44.733 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-06-09T21:59:44.805 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-06-09T21:59:44.871 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:44.871 INFO:tasks.ceph:role: osd.1 2024-06-09T21:59:44.871 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T21:59:44.872 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-09T21:59:44.937 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-09T21:59:44.937 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-09T21:59:44.937 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-09T21:59:44.937 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-09T21:59:44.937 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-09T21:59:44.937 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-06-09T21:59:44.937 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-09T21:59:44.937 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-09T21:59:44.938 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-09T21:59:44.938 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-09T21:59:44.980 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-06-09T21:59:44.983 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-06-09T21:59:44.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-06-09T21:59:45.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-06-09T21:59:45.124 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-06-09T21:59:45.191 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:45.191 INFO:tasks.ceph:role: osd.2 2024-06-09T21:59:45.191 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T21:59:45.191 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-09T21:59:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-09T21:59:45.258 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-09T21:59:45.258 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-09T21:59:45.258 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-09T21:59:45.258 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-09T21:59:45.258 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-06-09T21:59:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-09T21:59:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-09T21:59:45.259 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-09T21:59:45.259 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-09T21:59:45.297 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-06-09T21:59:45.299 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-06-09T21:59:45.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-06-09T21:59:45.369 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-06-09T21:59:45.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-06-09T21:59:45.505 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:45.505 INFO:tasks.ceph:role: osd.3 2024-06-09T21:59:45.506 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T21:59:45.506 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-09T21:59:45.571 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-09T21:59:45.571 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-09T21:59:45.571 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-09T21:59:45.571 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-09T21:59:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-09T21:59:45.572 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-06-09T21:59:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-09T21:59:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-09T21:59:45.572 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-09T21:59:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-09T21:59:45.610 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-06-09T21:59:45.612 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-06-09T21:59:45.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-06-09T21:59:45.682 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-06-09T21:59:45.751 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-09T21:59:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:45.883+0000 7fcc2d7a3640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-06-09T21:59:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:45.883+0000 7fcc2d7a3640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-06-09T21:59:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:45.883+0000 7fcc2d7a3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-09T21:59:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:45.883+0000 7fcc2d7a3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-09T21:59:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:45.883+0000 7fcc2d7a3640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-09T21:59:45.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:45.884+0000 7fcc2d7a3640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-06-09T21:59:48.714 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-09T21:59:48.745 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-09T21:59:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:48.853+0000 7f8fe3e24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-06-09T21:59:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:48.853+0000 7f8fe3e24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-06-09T21:59:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:48.853+0000 7f8fe3e24640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-09T21:59:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:48.853+0000 7f8fe3e24640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-09T21:59:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:48.853+0000 7f8fe3e24640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-09T21:59:48.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:48.854+0000 7f8fe3e24640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-06-09T21:59:51.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-09T21:59:51.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-09T21:59:51.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:51.835+0000 7f2ee9a25640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-06-09T21:59:51.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:51.835+0000 7f2ee9a25640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-06-09T21:59:51.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:51.835+0000 7f2ee9a25640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-09T21:59:51.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:51.835+0000 7f2ee9a25640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-09T21:59:51.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:51.835+0000 7f2ee9a25640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-09T21:59:51.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:51.836+0000 7f2ee9a25640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-06-09T21:59:54.665 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-09T21:59:54.696 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-09T21:59:54.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:54.803+0000 7f37a5f01640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-06-09T21:59:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:54.803+0000 7f37a5f01640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-06-09T21:59:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:54.803+0000 7f37a5f01640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-09T21:59:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:54.803+0000 7f37a5f01640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-09T21:59:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:54.803+0000 7f37a5f01640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-09T21:59:54.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T21:59:54.804+0000 7f37a5f01640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-06-09T21:59:57.630 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-09T21:59:57.661 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-06-09T21:59:57.691 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:57.691 INFO:tasks.ceph:role: osd.4 2024-06-09T21:59:57.691 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T21:59:57.691 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-09T21:59:57.768 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-09T21:59:57.768 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-09T21:59:57.769 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-09T21:59:57.769 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-09T21:59:57.769 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-09T21:59:57.769 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-06-09T21:59:57.769 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-09T21:59:57.769 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-09T21:59:57.769 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-09T21:59:57.769 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-09T21:59:58.277 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-06-09T21:59:58.280 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-06-09T21:59:58.280 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-06-09T21:59:58.421 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-06-09T21:59:58.456 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-06-09T21:59:58.525 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:58.525 INFO:tasks.ceph:role: osd.5 2024-06-09T21:59:58.525 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T21:59:58.525 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-09T21:59:58.599 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-09T21:59:58.599 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-09T21:59:58.599 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-09T21:59:58.599 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-09T21:59:58.599 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-09T21:59:58.599 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-06-09T21:59:58.600 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-09T21:59:58.600 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-09T21:59:58.600 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-09T21:59:58.600 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-09T21:59:59.046 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-06-09T21:59:59.048 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-06-09T21:59:59.048 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-06-09T21:59:59.139 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-06-09T21:59:59.171 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-06-09T21:59:59.238 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-09T21:59:59.238 INFO:tasks.ceph:role: osd.6 2024-06-09T21:59:59.239 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T21:59:59.239 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-09T21:59:59.313 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-09T21:59:59.314 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-09T21:59:59.803 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-06-09T21:59:59.805 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-06-09T21:59:59.805 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-06-09T21:59:59.923 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-06-09T21:59:59.956 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-06-09T22:00:00.024 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-09T22:00:00.024 INFO:tasks.ceph:role: osd.7 2024-06-09T22:00:00.024 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T22:00:00.024 DEBUG:teuthology.orchestra.run.smithi148:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-09T22:00:00.100 INFO:teuthology.orchestra.run.smithi148.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-09T22:00:00.100 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-09T22:00:00.100 INFO:teuthology.orchestra.run.smithi148.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-09T22:00:00.100 INFO:teuthology.orchestra.run.smithi148.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-09T22:00:00.100 INFO:teuthology.orchestra.run.smithi148.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-09T22:00:00.100 INFO:teuthology.orchestra.run.smithi148.stdout: = sunit=0 swidth=0 blks 2024-06-09T22:00:00.100 INFO:teuthology.orchestra.run.smithi148.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-09T22:00:00.100 INFO:teuthology.orchestra.run.smithi148.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-09T22:00:00.101 INFO:teuthology.orchestra.run.smithi148.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-09T22:00:00.101 INFO:teuthology.orchestra.run.smithi148.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-09T22:00:00.601 INFO:teuthology.orchestra.run.smithi148.stdout:Discarding blocks...Done. 2024-06-09T22:00:00.672 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi148.front.sepia.ceph.com -o noatime 2024-06-09T22:00:00.672 DEBUG:teuthology.orchestra.run.smithi148:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-06-09T22:00:00.807 DEBUG:teuthology.orchestra.run.smithi148:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-06-09T22:00:00.845 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-09T22:00:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:00.963+0000 7fd07c533640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-06-09T22:00:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:00.963+0000 7fd07c533640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-06-09T22:00:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:00.963+0000 7fd07c533640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-09T22:00:00.948 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:00.963+0000 7fd07c533640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-09T22:00:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:00.963+0000 7fd07c533640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-09T22:00:00.949 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:00.965+0000 7fd07c533640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-06-09T22:00:03.807 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-09T22:00:03.837 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-09T22:00:03.930 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:03.946+0000 7f9cc3a1e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-06-09T22:00:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:03.946+0000 7f9cc3a1e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-06-09T22:00:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:03.946+0000 7f9cc3a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-09T22:00:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:03.946+0000 7f9cc3a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-09T22:00:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:03.946+0000 7f9cc3a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-09T22:00:03.931 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:03.947+0000 7f9cc3a1e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-06-09T22:00:06.786 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-09T22:00:06.816 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-09T22:00:06.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:06.922+0000 7f4d772b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-06-09T22:00:06.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:06.922+0000 7f4d772b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-06-09T22:00:06.906 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:06.922+0000 7f4d772b5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-09T22:00:06.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:06.922+0000 7f4d772b5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-09T22:00:06.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:06.922+0000 7f4d772b5640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-09T22:00:06.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:06.923+0000 7f4d772b5640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-06-09T22:00:09.755 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-09T22:00:09.783 DEBUG:teuthology.orchestra.run.smithi148:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-09T22:00:09.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:09.890+0000 7f289e219640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-06-09T22:00:09.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:09.891+0000 7f289e219640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-06-09T22:00:09.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:09.891+0000 7f289e219640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-09T22:00:09.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:09.891+0000 7f289e219640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-09T22:00:09.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:09.891+0000 7f289e219640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-09T22:00:09.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:09.891+0000 7f289e219640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-06-09T22:00:11.330 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:00:11.333 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:00:12.723 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-09T22:00:12.753 INFO:tasks.ceph:Reading keys from all nodes... 2024-06-09T22:00:12.753 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:12.753 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-06-09T22:00:12.785 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:12.785 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-06-09T22:00:12.854 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:12.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-06-09T22:00:12.924 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:12.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-06-09T22:00:12.992 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:12.992 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-06-09T22:00:13.062 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:13.062 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-06-09T22:00:13.130 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:13.130 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-06-09T22:00:13.197 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:13.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-06-09T22:00:13.263 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:13.263 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-06-09T22:00:13.294 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:13.294 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-06-09T22:00:13.361 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:13.362 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-06-09T22:00:13.428 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:13.429 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-06-09T22:00:13.495 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:13.495 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-06-09T22:00:13.562 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:13.563 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-06-09T22:00:13.629 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:13.630 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-06-09T22:00:13.695 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:13.695 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-06-09T22:00:13.761 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:13.762 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-06-09T22:00:13.781 INFO:tasks.ceph:Adding keys to all mons... 2024-06-09T22:00:13.781 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-09T22:00:13.824 DEBUG:teuthology.orchestra.run.smithi148:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.y] 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDPJWZmYBgsLhAAuCXFrFk5LJLUj2MgOTNPPw== 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.a] 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDPJWZmQkemNRAAdkQ9xbtYDbHf1aC6w8VgEw== 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.c] 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDQJWZm6foyARAAJBIH5ep0fthbGqOfO2BTUA== 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.e] 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDQJWZmce5XCBAAiMZk+tFqTUW3Yiadzy6jFg== 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.0] 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDRJWZmF125NBAAB2vLCeIN4UqmkXANBiH73A== 2024-06-09T22:00:13.848 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.1] 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDUJWZmdGLwMhAAoEwAbIs70jYFfaLly+IMBg== 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.2] 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDXJWZmX+HbMRAAN/xCt8QUBV6kOrVWEabT9w== 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.3] 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDaJWZmR+vzLxAApw4cBwprQ/U6WvzVUePqgw== 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout:[mgr.x] 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDPJWZmME5WMhAAU6oMVJP9a4z7jLONU72JlQ== 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.b] 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDQJWZmz4cTDRAApErjuFBAnft5vhVVFwe74Q== 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.d] 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDQJWZmr0s4FBAA+8FsRmSNrfS4T/dif0XglA== 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout:[mds.f] 2024-06-09T22:00:13.849 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDQJWZmwfhuGxAAPGa78LoL66G5UwVfXa16vQ== 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.4] 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDgJWZm+Ux5ORAAQeLnrMX2qVjIrIQVKZVVLQ== 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.5] 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDjJWZmxM92OBAAuy3YrDcxML7UXQjK5kfC7g== 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.6] 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDmJWZmoPMENxAABCzhH/CnwV39aG1rPRi6SQ== 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout:[osd.7] 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDpJWZmPc8kNRAAOXB9zrDALPvDd770Ei554Q== 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout:[client.0] 2024-06-09T22:00:13.850 INFO:teuthology.orchestra.run.smithi148.stdout: key = AQDQJWZmlufCHxAA1Jx0gHVqEnkcK97VpaAO4A== 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.y] 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDPJWZmYBgsLhAAuCXFrFk5LJLUj2MgOTNPPw== 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a] 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDPJWZmQkemNRAAdkQ9xbtYDbHf1aC6w8VgEw== 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c] 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDQJWZm6foyARAAJBIH5ep0fthbGqOfO2BTUA== 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.e] 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDQJWZmce5XCBAAiMZk+tFqTUW3Yiadzy6jFg== 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDRJWZmF125NBAAB2vLCeIN4UqmkXANBiH73A== 2024-06-09T22:00:13.852 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDUJWZmdGLwMhAAoEwAbIs70jYFfaLly+IMBg== 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDXJWZmX+HbMRAAN/xCt8QUBV6kOrVWEabT9w== 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.3] 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDaJWZmR+vzLxAApw4cBwprQ/U6WvzVUePqgw== 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDPJWZmME5WMhAAU6oMVJP9a4z7jLONU72JlQ== 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b] 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDQJWZmz4cTDRAApErjuFBAnft5vhVVFwe74Q== 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d] 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDQJWZmr0s4FBAA+8FsRmSNrfS4T/dif0XglA== 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.f] 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDQJWZmwfhuGxAAPGa78LoL66G5UwVfXa16vQ== 2024-06-09T22:00:13.853 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.4] 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDgJWZm+Ux5ORAAQeLnrMX2qVjIrIQVKZVVLQ== 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.5] 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDjJWZmxM92OBAAuy3YrDcxML7UXQjK5kfC7g== 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.6] 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDmJWZmoPMENxAABCzhH/CnwV39aG1rPRi6SQ== 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.7] 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDpJWZmPc8kNRAAOXB9zrDALPvDd770Ei554Q== 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-06-09T22:00:13.854 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQDQJWZmlufCHxAA1Jx0gHVqEnkcK97VpaAO4A== 2024-06-09T22:00:13.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-09T22:00:13.894 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-09T22:00:13.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:13.953 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.010 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.066 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.120 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.123 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.174 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.176 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.227 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.229 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.280 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.283 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.333 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-09T22:00:14.336 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-09T22:00:14.386 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.388 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.442 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.444 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.500 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-06-09T22:00:14.554 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.556 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.607 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.609 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.662 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.664 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.719 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.721 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-09T22:00:14.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-09T22:00:14.775 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-09T22:00:14.828 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-06-09T22:00:14.828 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-06-09T22:00:14.854 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-09T22:00:14.951 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-06-09T22:00:14.976 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-06-09T22:00:15.002 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-09T22:00:15.098 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-06-09T22:00:15.123 DEBUG:teuthology.orchestra.run.smithi148:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-06-09T22:00:15.187 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-09T22:00:15.283 DEBUG:teuthology.orchestra.run.smithi148:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-06-09T22:00:15.308 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-09T22:00:15.311 DEBUG:teuthology.orchestra.run.smithi148:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-09T22:00:15.364 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-06-09T22:00:15.364 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-09T22:00:15.364 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-09T22:00:15.364 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-06-09T22:00:15.367 INFO:tasks.ceph.mon.a:Started 2024-06-09T22:00:15.367 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-09T22:00:15.367 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-09T22:00:15.367 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-06-09T22:00:15.406 INFO:tasks.ceph.mon.b:Started 2024-06-09T22:00:15.406 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-09T22:00:15.406 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-09T22:00:15.407 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-06-09T22:00:15.409 INFO:tasks.ceph.mon.c:Started 2024-06-09T22:00:15.409 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-06-09T22:00:15.409 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-06-09T22:00:15.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-06-09T22:00:15.412 INFO:tasks.ceph.mgr.y:Started 2024-06-09T22:00:15.412 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-06-09T22:00:15.413 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-06-09T22:00:15.415 INFO:tasks.ceph.mgr.x:Started 2024-06-09T22:00:15.415 INFO:tasks.ceph:Setting crush tunables to default 2024-06-09T22:00:15.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd crush tunables default 2024-06-09T22:00:15.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.536+0000 7fab32da5640 1 Processor -- start 2024-06-09T22:00:15.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.536+0000 7fab32da5640 1 -- start start 2024-06-09T22:00:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.537+0000 7fab32da5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab2c0cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.537+0000 7fab32da5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2c15a9f0 con 0x7fab2c0cf230 2024-06-09T22:00:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.537+0000 7fab32da5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2c15acb0 con 0x7fab2c0ccbc0 2024-06-09T22:00:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.537+0000 7fab32da5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7fab2c15af90 con 0x7fab2c0ce7a0 2024-06-09T22:00:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.537+0000 7fab2bfff640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab2c0ce090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-06-09T22:00:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.537+0000 7fab31342640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab2c0cf680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2024-06-09T22:00:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.537+0000 7fab31342640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab2c0cf680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:00:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.537+0000 7fab30b41640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fab2c0ccbc0 legacy=0x7fab2c0ca6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-06-09T22:00:15.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.738+0000 7fab2bfff640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab2c0ce090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-06-09T22:00:15.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.738+0000 7fab30b41640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fab2c0ccbc0 legacy=0x7fab2c0ca6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-06-09T22:00:15.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.738+0000 7fab31342640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab2c0cf680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2024-06-09T22:00:15.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:15.738+0000 7fab31342640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab2c0cf680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:00:16.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:16.139+0000 7fab2bfff640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab2c0ce090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-06-09T22:00:16.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:16.139+0000 7fab31342640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab2c0cf680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2024-06-09T22:00:16.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:16.139+0000 7fab30b41640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fab2c0ccbc0 legacy=0x7fab2c0ca6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-06-09T22:00:16.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:16.139+0000 7fab31342640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab2c0cf680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:00:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:16.940+0000 7fab2bfff640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab2c0ce090 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-06-09T22:00:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:16.940+0000 7fab31342640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab2c0cf680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2024-06-09T22:00:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:16.940+0000 7fab30b41640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fab2c0ccbc0 legacy=0x7fab2c0ca6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.148:6789/0 2024-06-09T22:00:16.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:16.940+0000 7fab31342640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab2c0cf680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2a7fc640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab2c0cf680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2a7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab2c0cf680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2a7fc640 1 -- >> v1:172.21.15.148:6789/0 conn(0x7fab2c0ccbc0 legacy=0x7fab2c0ca6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2a7fc640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab2c0ce090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2a7fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 0x7fab10005b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2a7fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2a7fc640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7fab2c15af90 con 0x7fab100036a0 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2a7fc640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fab10008d30 con 0x7fab100061a0 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab30b41640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 msgr2=0x7fab10005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab30b41640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 0x7fab10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2bfff640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2024-06-09T22:00:18.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.537+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:00:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.738+0000 7fab30b41640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 msgr2=0x7fab10005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-06-09T22:00:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.738+0000 7fab30b41640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 0x7fab10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:00:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.738+0000 7fab2bfff640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2024-06-09T22:00:18.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:18.738+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:00:19.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:19.139+0000 7fab30b41640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 msgr2=0x7fab10005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-06-09T22:00:19.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:19.139+0000 7fab30b41640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 0x7fab10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:00:19.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:19.139+0000 7fab2bfff640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2024-06-09T22:00:19.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:19.139+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:00:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:19.940+0000 7fab30b41640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 msgr2=0x7fab10005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.32:3300/0 2024-06-09T22:00:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:19.940+0000 7fab30b41640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 0x7fab10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:00:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:19.940+0000 7fab2bfff640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2024-06-09T22:00:19.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:19.940+0000 7fab2bfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:00:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 -- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 msgr2=0x7fab10005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab100036a0 0x7fab10005b20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab31342640 1 -- reap_dead start 2024-06-09T22:00:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab1000e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:21.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2c15b270 con 0x7fab2c0cf230 2024-06-09T22:00:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7fab2c15af90 con 0x7fab2c0ce7a0 2024-06-09T22:00:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2a7fc640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fab10008d30 con 0x7fab100061a0 2024-06-09T22:00:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab2bfff640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab1000b120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-06-09T22:00:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab31342640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2024-06-09T22:00:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab30b41640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2024-06-09T22:00:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab31342640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab1000e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:00:21.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.536+0000 7fab30b41640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:00:21.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.737+0000 7fab2bfff640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab1000b120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-06-09T22:00:21.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.737+0000 7fab31342640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2024-06-09T22:00:21.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.737+0000 7fab30b41640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2024-06-09T22:00:21.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.737+0000 7fab31342640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab1000e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:00:21.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:21.737+0000 7fab30b41640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:00:22.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.138+0000 7fab2bfff640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab1000b120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-06-09T22:00:22.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.138+0000 7fab30b41640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2024-06-09T22:00:22.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.138+0000 7fab31342640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2024-06-09T22:00:22.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.138+0000 7fab30b41640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:00:22.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.138+0000 7fab31342640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab1000e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:00:22.702 INFO:tasks.ceph.mon.b.smithi148.stderr:==00:00:00:07.163 37377== Warning: unimplemented fcntl command: 1036 2024-06-09T22:00:22.703 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:00:07.165 37378== Warning: unimplemented fcntl command: 1036 2024-06-09T22:00:22.821 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:00:07.334 37657== Warning: unimplemented fcntl command: 1036 2024-06-09T22:00:22.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.939+0000 7fab2bfff640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab1000b120 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-06-09T22:00:22.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.939+0000 7fab30b41640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3301/0 2024-06-09T22:00:22.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.939+0000 7fab31342640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.148:3300/0 2024-06-09T22:00:22.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.939+0000 7fab31342640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab1000e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:00:22.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:22.939+0000 7fab30b41640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:00:23.022 INFO:tasks.ceph.mon.b.smithi148.stderr:==00:00:00:07.484 37377== Warning: unimplemented fcntl command: 1036 2024-06-09T22:00:23.023 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:00:07.485 37378== Warning: unimplemented fcntl command: 1036 2024-06-09T22:00:23.148 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:00:07.661 37657== Warning: unimplemented fcntl command: 1036 2024-06-09T22:00:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.536+0000 7fab2a7fc640 1 -- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 msgr2=0x7fab10008660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.536+0000 7fab2a7fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab100061a0 0x7fab10008660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.536+0000 7fab2a7fc640 1 -- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.536+0000 7fab2a7fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fab2c0ce7a0 0x7fab1000e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.536+0000 7fab2a7fc640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fab2c0cf230 legacy=0x7fab1000b120 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:24.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.536+0000 7fab2a7fc640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.536+0000 7fab2a7fc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2c15a9f0 con 0x7fab1000f410 2024-06-09T22:00:24.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.536+0000 7fab2a7fc640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7fab2c15af90 con 0x7fab10003b80 2024-06-09T22:00:24.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.654+0000 7fab30b41640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:24.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.675+0000 7fab30b41640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43238/0 (socket says 172.21.15.32:43238) 2024-06-09T22:00:24.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.675+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 learned_addr learned my addr 172.21.15.32:0/3407252610 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.711+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-09T22:00:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.711+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-09T22:00:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.711+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:00:24.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.711+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:00:24.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.913+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.914+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-09T22:00:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.914+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-09T22:00:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.914+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:00:24.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:24.914+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:00:25.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:25.602+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:25.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:25.603+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-09T22:00:25.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:25.603+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-09T22:00:25.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:25.603+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:00:25.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:25.603+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-09T22:00:27.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.049+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.051+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-09T22:00:27.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.051+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-09T22:00:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.051+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:00:27.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.051+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-09T22:00:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab2a7fc640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab2a7fc640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab2a7fc640 1 -- 172.21.15.32:0/3407252610 >> v1:172.21.15.148:6789/0 conn(0x7fab1000f410 legacy=0x7fab10011800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab2a7fc640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab31342640 1 -- 172.21.15.32:0/3407252610 reap_dead start 2024-06-09T22:00:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab2a7fc640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 0x7fab1000e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:27.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab2a7fc640 1 -- 172.21.15.32:0/3407252610 --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2c15b270 con 0x7fab10013090 2024-06-09T22:00:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab2a7fc640 1 -- 172.21.15.32:0/3407252610 --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7fab2c15af90 con 0x7fab10003b80 2024-06-09T22:00:27.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.536+0000 7fab2a7fc640 1 -- 172.21.15.32:0/3407252610 --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7fab10008d30 con 0x7fab10013c20 2024-06-09T22:00:27.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.554+0000 7fab31342640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 0x7fab1000e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:27.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.557+0000 7fab31342640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-09T22:00:27.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.557+0000 7fab31342640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-09T22:00:27.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.557+0000 7fab31342640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 0x7fab1000e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:00:27.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.557+0000 7fab31342640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 0x7fab1000e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:00:27.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.560+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:27.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.561+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-09T22:00:27.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.561+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-09T22:00:27.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.561+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:00:27.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.561+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-09T22:00:27.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.887+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:27.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.892+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-06-09T22:00:27.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.893+0000 7fab30b41640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-09T22:00:27.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.893+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:00:27.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.893+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:00:27.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:27.899+0000 7fab31342640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 0x7fab1000e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:28.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.070+0000 7fab31342640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-06-09T22:00:28.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.070+0000 7fab31342640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-06-09T22:00:28.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.070+0000 7fab31342640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 0x7fab1000e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-06-09T22:00:28.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.070+0000 7fab31342640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 0x7fab1000e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-09T22:00:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.297+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 <== mon.1 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818774729 0 0) 0x7fab2c15b270 con 0x7fab10013090 2024-06-09T22:00:28.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.297+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fab1c003520 con 0x7fab10013090 2024-06-09T22:00:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.331+0000 7fab30b41640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 <== mon.1 v1:172.21.15.148:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 896829176 0 0) 0x7fab1c003520 con 0x7fab10013090 2024-06-09T22:00:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 msgr2=0x7fab1000e8e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-09T22:00:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.148:3300/0 conn(0x7fab10013c20 0x7fab1000e8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 msgr2=0x7fab1000eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 --2- 172.21.15.32:0/3407252610 >> v2:172.21.15.32:3300/0 conn(0x7fab10003b80 0x7fab1000eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 --> v1:172.21.15.148:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c15acb0 con 0x7fab10013090 2024-06-09T22:00:28.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 <== mon.1 v1:172.21.15.148:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 909852050 0 0) 0x7fab180040a0 con 0x7fab10013090 2024-06-09T22:00:28.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 >> v1:172.21.15.148:6789/0 conn(0x7fab10013090 legacy=0x7fab10013470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:28.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab1c0039b0 0x7fab1c003dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:28.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab1c004c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:28.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab1c0051c0 0x7fab1c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:28.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c15af90 con 0x7fab1c004800 2024-06-09T22:00:28.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fab10008d30 con 0x7fab1c0051c0 2024-06-09T22:00:28.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.438+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fab1c00c060 con 0x7fab1c0039b0 2024-06-09T22:00:28.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.452+0000 7fab2bfff640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab1c004c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.454+0000 7fab2bfff640 1 -- 172.21.15.32:0/3407252610 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab1c0039b0 msgr2=0x7fab1c003dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:28.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.454+0000 7fab2bfff640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab1c0039b0 0x7fab1c003dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:28.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.454+0000 7fab2bfff640 1 -- 172.21.15.32:0/3407252610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab1c0051c0 msgr2=0x7fab1c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:28.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.454+0000 7fab2bfff640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab1c0051c0 0x7fab1c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:28.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.454+0000 7fab2bfff640 1 -- 172.21.15.32:0/3407252610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c003520 con 0x7fab1c004800 2024-06-09T22:00:28.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.454+0000 7fab31342640 1 -- 172.21.15.32:0/3407252610 reap_dead start 2024-06-09T22:00:28.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.465+0000 7fab2bfff640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab1c004c80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fab180038b0 tx=0x7fab1c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6feaeeeb926ad94 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:28.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.477+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab18007cb0 con 0x7fab1c004800 2024-06-09T22:00:28.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.477+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab18032430 con 0x7fab1c004800 2024-06-09T22:00:28.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.477+0000 7fab2b7fe640 1 -- 172.21.15.32:0/3407252610 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab18032820 con 0x7fab1c004800 2024-06-09T22:00:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.478+0000 7fab32da5640 1 -- 172.21.15.32:0/3407252610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 msgr2=0x7fab1c004c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.478+0000 7fab32da5640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab1c004c80 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fab180038b0 tx=0x7fab1c005830 comp rx=0 tx=0).stop 2024-06-09T22:00:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.478+0000 7fab32da5640 1 -- 172.21.15.32:0/3407252610 shutdown_connections 2024-06-09T22:00:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.478+0000 7fab32da5640 1 --2- 172.21.15.32:0/3407252610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab1c004c80 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.478+0000 7fab32da5640 1 -- 172.21.15.32:0/3407252610 >> 172.21.15.32:0/3407252610 conn(0x7fab2c0b1880 msgr2=0x7fab2c0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:28.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.478+0000 7fab32da5640 1 -- 172.21.15.32:0/3407252610 shutdown_connections 2024-06-09T22:00:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.478+0000 7fab32da5640 1 -- 172.21.15.32:0/3407252610 wait complete. 2024-06-09T22:00:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.479+0000 7fab32da5640 1 Processor -- start 2024-06-09T22:00:28.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.480+0000 7fab32da5640 1 -- start start 2024-06-09T22:00:28.464 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.480+0000 7fab32da5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab2c0c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.480+0000 7fab32da5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0c90c0 0x7fab2c1e3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.480+0000 7fab32da5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab2c0b46f0 0x7fab2c1e4070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.480+0000 7fab32da5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c15a6e0 con 0x7fab1c004800 2024-06-09T22:00:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.480+0000 7fab32da5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c0cdb70 con 0x7fab2c0c90c0 2024-06-09T22:00:28.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.480+0000 7fab32da5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fab2c0b52b0 con 0x7fab2c0b46f0 2024-06-09T22:00:28.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.496+0000 7fab31342640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab2c0b46f0 0x7fab2c1e4070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:28.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.497+0000 7fab31342640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab2c0b46f0 0x7fab2c1e4070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.32:39396/0 (socket says 172.21.15.32:39396) 2024-06-09T22:00:28.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.497+0000 7fab31342640 1 -- 172.21.15.32:0/4106207621 learned_addr learned my addr 172.21.15.32:0/4106207621 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:28.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.498+0000 7fab2bfff640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0c90c0 0x7fab2c1e3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:28.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.504+0000 7fab30b41640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab2c0c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.510+0000 7fab31342640 1 -- 172.21.15.32:0/4106207621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0c90c0 msgr2=0x7fab2c1e3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.510+0000 7fab31342640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0c90c0 0x7fab2c1e3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:28.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.510+0000 7fab31342640 1 -- 172.21.15.32:0/4106207621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 msgr2=0x7fab2c0c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:28.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.510+0000 7fab31342640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab2c0c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:28.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.510+0000 7fab31342640 1 -- 172.21.15.32:0/4106207621 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c1e4f00 con 0x7fab2c0b46f0 2024-06-09T22:00:28.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.514+0000 7fab31342640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab2c0b46f0 0x7fab2c1e4070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0cbf20 tx=0x7fab2400ff10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.519+0000 7fab297fa640 1 -- 172.21.15.32:0/4106207621 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24018440 con 0x7fab2c0b46f0 2024-06-09T22:00:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.519+0000 7fab297fa640 1 -- 172.21.15.32:0/4106207621 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab240185e0 con 0x7fab2c0b46f0 2024-06-09T22:00:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.519+0000 7fab297fa640 1 -- 172.21.15.32:0/4106207621 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab24018990 con 0x7fab2c0b46f0 2024-06-09T22:00:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.519+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2c1e51e0 con 0x7fab2c0b46f0 2024-06-09T22:00:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.519+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2c1f0cf0 con 0x7fab2c0b46f0 2024-06-09T22:00:28.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.520+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2c0b5460 con 0x7fab2c0b46f0 2024-06-09T22:00:28.641 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:28.656+0000 7fa707bb8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:00:28.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.748+0000 7fab297fa640 1 -- 172.21.15.32:0/4106207621 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fab24022410 con 0x7fab2c0b46f0 2024-06-09T22:00:28.737 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:28.753+0000 7f4ccb930040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-09T22:00:28.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.789+0000 7fab297fa640 1 -- 172.21.15.32:0/4106207621 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fab240219b0 con 0x7fab2c0b46f0 2024-06-09T22:00:28.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.791+0000 7fab297fa640 1 -- 172.21.15.32:0/4106207621 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fab2401e060 con 0x7fab2c0b46f0 2024-06-09T22:00:28.810 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:28.826+0000 7fa707bb8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:00:28.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:28.861+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fab2c068490 con 0x7fab2c0b46f0 2024-06-09T22:00:28.868 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:28.884+0000 7fa707bb8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:00:28.917 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:28.933+0000 7f4ccb930040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:00:28.969 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:28.985+0000 7f4ccb930040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-09T22:00:29.126 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:29.142+0000 7fa707bb8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:00:29.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.176+0000 7fab297fa640 1 -- 172.21.15.32:0/4106207621 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7fab24021bb0 con 0x7fab2c0b46f0 2024-06-09T22:00:29.160 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.176+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab2c0b46f0 msgr2=0x7fab2c1e4070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.176+0000 7fab32da5640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab2c0b46f0 0x7fab2c1e4070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fab2c0cbf20 tx=0x7fab2400ff10 comp rx=0 tx=0).stop 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.177+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 shutdown_connections 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.177+0000 7fab32da5640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fab2c0b46f0 0x7fab2c1e4070 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.177+0000 7fab32da5640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fab2c0c90c0 0x7fab2c1e3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.177+0000 7fab32da5640 1 --2- 172.21.15.32:0/4106207621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab1c004800 0x7fab2c0c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.177+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 >> 172.21.15.32:0/4106207621 conn(0x7fab2c0b1880 msgr2=0x7fab2c158dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.177+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 shutdown_connections 2024-06-09T22:00:29.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:29.177+0000 7fab32da5640 1 -- 172.21.15.32:0/4106207621 wait complete. 2024-06-09T22:00:29.172 INFO:tasks.ceph:check_enable_crimson: False 2024-06-09T22:00:29.173 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-06-09T22:00:29.173 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:29.173 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-06-09T22:00:29.182 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:29.198+0000 7fa707bb8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:00:29.199 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:29.199 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-06-09T22:00:29.224 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:29.241+0000 7f4ccb930040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-09T22:00:29.267 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:29.267 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-06-09T22:00:29.274 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:29.290+0000 7f4ccb930040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-09T22:00:29.332 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-06-09T22:00:29.332 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-06-09T22:00:29.397 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:29.397 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-06-09T22:00:29.422 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:29.422 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-06-09T22:00:29.487 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:29.487 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-06-09T22:00:29.552 DEBUG:teuthology.orchestra.run.smithi148:> set -ex 2024-06-09T22:00:29.553 DEBUG:teuthology.orchestra.run.smithi148:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-06-09T22:00:29.617 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new 4cc9d666-a633-4740-a293-be07ab208937 0 2024-06-09T22:00:29.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.774+0000 7f23f759e640 1 Processor -- start 2024-06-09T22:00:29.758 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.774+0000 7f23f759e640 1 -- start start 2024-06-09T22:00:29.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.775+0000 7f23f759e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f0154620 0x7f23f015a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:29.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.775+0000 7f23f759e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23f0072de0 0x7f23f0073290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:29.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.775+0000 7f23f759e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23f006e320 con 0x7f23f006bfb0 2024-06-09T22:00:29.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.775+0000 7f23f759e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f23f00710d0 con 0x7f23f0072de0 2024-06-09T22:00:29.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.775+0000 7f23f759e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f23f00737d0 con 0x7f23f0154620 2024-06-09T22:00:29.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.775+0000 7f23f659c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f0154620 0x7f23f015a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:29.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.776+0000 7f23f6d9d640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f23f0072de0 0x7f23f0073290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:29.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.776+0000 7f23f659c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f0154620 0x7f23f015a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:44344/0 (socket says 172.21.15.148:44344) 2024-06-09T22:00:29.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.776+0000 7f23f659c640 1 -- 172.21.15.148:0/3621861149 learned_addr learned my addr 172.21.15.148:0/3621861149 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:29.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.778+0000 7f23f659c640 1 -- 172.21.15.148:0/3621861149 >> v2:172.21.15.32:3300/0 conn(0x7f23f0072de0 msgr2=0x7f23f0073290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:29.762 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.778+0000 7f23f659c640 1 --2- 172.21.15.148:0/3621861149 >> v2:172.21.15.32:3300/0 conn(0x7f23f0072de0 0x7f23f0073290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:29.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.778+0000 7f23f659c640 1 -- 172.21.15.148:0/3621861149 >> v1:172.21.15.32:6789/0 conn(0x7f23f006bfb0 legacy=0x7f23f006b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:29.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.778+0000 7f23f659c640 1 -- 172.21.15.148:0/3621861149 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f006e600 con 0x7f23f0154620 2024-06-09T22:00:29.763 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.780+0000 7f23f659c640 1 --2- 172.21.15.148:0/3621861149 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f0154620 0x7f23f015a410 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009870 tx=0x7f23e402a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7bdd0153e5922594 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:29.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.783+0000 7f23f559a640 1 -- 172.21.15.148:0/3621861149 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4037070 con 0x7f23f0154620 2024-06-09T22:00:29.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.783+0000 7f23f559a640 1 -- 172.21.15.148:0/3621861149 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e402ae50 con 0x7f23f0154620 2024-06-09T22:00:29.767 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.783+0000 7f23f559a640 1 -- 172.21.15.148:0/3621861149 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4032860 con 0x7f23f0154620 2024-06-09T22:00:29.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.784+0000 7f23f759e640 1 -- 172.21.15.148:0/3621861149 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f0154620 msgr2=0x7f23f015a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:29.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.784+0000 7f23f759e640 1 --2- 172.21.15.148:0/3621861149 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f0154620 0x7f23f015a410 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f23e4009870 tx=0x7f23e402a300 comp rx=0 tx=0).stop 2024-06-09T22:00:29.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.784+0000 7f23f759e640 1 -- 172.21.15.148:0/3621861149 shutdown_connections 2024-06-09T22:00:29.768 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.784+0000 7f23f759e640 1 --2- 172.21.15.148:0/3621861149 >> v2:172.21.15.32:3300/0 conn(0x7f23f0072de0 0x7f23f0073290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:29.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.784+0000 7f23f759e640 1 --2- 172.21.15.148:0/3621861149 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f0154620 0x7f23f015a410 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:29.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.784+0000 7f23f759e640 1 -- 172.21.15.148:0/3621861149 >> 172.21.15.148:0/3621861149 conn(0x7f23f0070440 msgr2=0x7f23f0070850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:29.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.784+0000 7f23f759e640 1 -- 172.21.15.148:0/3621861149 shutdown_connections 2024-06-09T22:00:29.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.784+0000 7f23f759e640 1 -- 172.21.15.148:0/3621861149 wait complete. 2024-06-09T22:00:29.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.785+0000 7f23f759e640 1 Processor -- start 2024-06-09T22:00:29.769 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.786+0000 7f23f759e640 1 -- start start 2024-06-09T22:00:29.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.786+0000 7f23f759e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f006bfb0 0x7f23f014b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:29.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.786+0000 7f23f759e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f0072de0 0x7f23f014bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:29.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.786+0000 7f23f759e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23f0154620 0x7f23f014cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:29.770 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.786+0000 7f23f759e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23f006eea0 con 0x7f23f0072de0 2024-06-09T22:00:29.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.786+0000 7f23f759e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f23f006ebc0 con 0x7f23f0154620 2024-06-09T22:00:29.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.786+0000 7f23f759e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f23f006dfc0 con 0x7f23f006bfb0 2024-06-09T22:00:29.771 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.787+0000 7f23f659c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f006bfb0 0x7f23f014b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:29.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.787+0000 7f23f6d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23f0154620 0x7f23f014cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:29.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.787+0000 7f23f5d9b640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f0072de0 0x7f23f014bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:29.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.787+0000 7f23f659c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f006bfb0 0x7f23f014b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:44354/0 (socket says 172.21.15.148:44354) 2024-06-09T22:00:29.772 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.787+0000 7f23f659c640 1 -- 172.21.15.148:0/314780294 learned_addr learned my addr 172.21.15.148:0/314780294 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:29.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.790+0000 7f23f659c640 1 -- 172.21.15.148:0/314780294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23f0154620 msgr2=0x7f23f014cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:29.774 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.790+0000 7f23f659c640 1 --2- 172.21.15.148:0/314780294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23f0154620 0x7f23f014cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:29.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.790+0000 7f23f659c640 1 -- 172.21.15.148:0/314780294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f0072de0 msgr2=0x7f23f014bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:29.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.790+0000 7f23f659c640 1 --2- 172.21.15.148:0/314780294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f0072de0 0x7f23f014bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:29.775 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.790+0000 7f23f659c640 1 -- 172.21.15.148:0/314780294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23f0152ff0 con 0x7f23f006bfb0 2024-06-09T22:00:29.776 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.793+0000 7f23f659c640 1 --2- 172.21.15.148:0/314780294 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f006bfb0 0x7f23f014b860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23e4000c00 tx=0x7f23e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:29.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.796+0000 7f23e2ffd640 1 -- 172.21.15.148:0/314780294 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4041070 con 0x7f23f006bfb0 2024-06-09T22:00:29.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.796+0000 7f23e2ffd640 1 -- 172.21.15.148:0/314780294 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23e403c680 con 0x7f23f006bfb0 2024-06-09T22:00:29.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.796+0000 7f23e2ffd640 1 -- 172.21.15.148:0/314780294 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e4037040 con 0x7f23f006bfb0 2024-06-09T22:00:29.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.796+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23f01532d0 con 0x7f23f006bfb0 2024-06-09T22:00:29.780 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.796+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23f0157c40 con 0x7f23f006bfb0 2024-06-09T22:00:29.781 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.797+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23f004f760 con 0x7f23f006bfb0 2024-06-09T22:00:29.883 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.899+0000 7f23e2ffd640 1 -- 172.21.15.148:0/314780294 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f23e4044070 con 0x7f23f006bfb0 2024-06-09T22:00:29.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.899+0000 7f23e2ffd640 1 -- 172.21.15.148:0/314780294 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f23e4046c30 con 0x7f23f006bfb0 2024-06-09T22:00:29.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:29.899+0000 7f23e2ffd640 1 -- 172.21.15.148:0/314780294 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f23e4050470 con 0x7f23f006bfb0 2024-06-09T22:00:29.988 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:30.004+0000 7fa707bb8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:00:29.992 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.008+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd new", "uuid": "4cc9d666-a633-4740-a293-be07ab208937", "id": 0} v 0) -- 0x7f23f00c8a70 con 0x7f23f006bfb0 2024-06-09T22:00:30.080 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:30.096+0000 7f4ccb930040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-09T22:00:30.081 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:30.098+0000 7fa707bb8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:00:30.130 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:30.146+0000 7fa707bb8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:00:30.154 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.170+0000 7f23e2ffd640 1 -- 172.21.15.148:0/314780294 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4cc9d666-a633-4740-a293-be07ab208937", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f23e4046450 con 0x7f23f006bfb0 2024-06-09T22:00:30.154 INFO:teuthology.orchestra.run.smithi148.stdout:0 2024-06-09T22:00:30.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f006bfb0 msgr2=0x7f23f014b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.155 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 --2- 172.21.15.148:0/314780294 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f006bfb0 0x7f23f014b860 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23e4000c00 tx=0x7f23e40046c0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 shutdown_connections 2024-06-09T22:00:30.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 --2- 172.21.15.148:0/314780294 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f23f0154620 0x7f23f014cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 --2- 172.21.15.148:0/314780294 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23f0072de0 0x7f23f014bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 --2- 172.21.15.148:0/314780294 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f23f006bfb0 0x7f23f014b860 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 >> 172.21.15.148:0/314780294 conn(0x7f23f0070440 msgr2=0x7f23f00cbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:30.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 shutdown_connections 2024-06-09T22:00:30.156 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.171+0000 7f23f759e640 1 -- 172.21.15.148:0/314780294 wait complete. 2024-06-09T22:00:30.170 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new 8db4805f-af3c-41a5-8eaf-1485c4ec8a5e 1 2024-06-09T22:00:30.180 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:30.196+0000 7f4ccb930040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-09T22:00:30.230 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:30.246+0000 7f4ccb930040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-09T22:00:30.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.293+0000 7f2d66963640 1 Processor -- start 2024-06-09T22:00:30.277 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.293+0000 7f2d66963640 1 -- start start 2024-06-09T22:00:30.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.294+0000 7f2d66963640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2d6015dd70 0x7f2d6006b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.294+0000 7f2d66963640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d6015e860 con 0x7f2d60073110 2024-06-09T22:00:30.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.294+0000 7f2d66963640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d600c8910 con 0x7f2d600c7620 2024-06-09T22:00:30.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.294+0000 7f2d66963640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7f2d600c8bf0 con 0x7f2d6015dd70 2024-06-09T22:00:30.278 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.294+0000 7f2d5ffff640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2d6015dd70 0x7f2d6006b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.294+0000 7f2d64f00640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2d60073110 0x7f2d60154330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.148:45736/0 (socket says 172.21.15.148:45736) 2024-06-09T22:00:30.279 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.294+0000 7f2d64f00640 1 -- 172.21.15.148:0/621078638 learned_addr learned my addr 172.21.15.148:0/621078638 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-06-09T22:00:30.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.297+0000 7f2d5ffff640 1 -- 172.21.15.148:0/621078638 >> v1:172.21.15.148:6789/0 conn(0x7f2d600c7620 legacy=0x7f2d6015e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:30.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.297+0000 7f2d5ffff640 1 -- 172.21.15.148:0/621078638 >> v1:172.21.15.32:6789/0 conn(0x7f2d60073110 legacy=0x7f2d60154330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.281 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.297+0000 7f2d5ffff640 1 -- 172.21.15.148:0/621078638 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d60072350 con 0x7f2d6015dd70 2024-06-09T22:00:30.283 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.299+0000 7f2d5ffff640 1 --2- 172.21.15.148:0/621078638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d6015dd70 0x7f2d6006b390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2d500097a0 tx=0x7f2d5002a1e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=82fa670aff1ebc13 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:30.289 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.305+0000 7f2d5effd640 1 -- 172.21.15.148:0/621078638 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50037070 con 0x7f2d6015dd70 2024-06-09T22:00:30.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.305+0000 7f2d5effd640 1 -- 172.21.15.148:0/621078638 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d50032440 con 0x7f2d6015dd70 2024-06-09T22:00:30.290 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.305+0000 7f2d5effd640 1 -- 172.21.15.148:0/621078638 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d50032810 con 0x7f2d6015dd70 2024-06-09T22:00:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.306+0000 7f2d66963640 1 -- 172.21.15.148:0/621078638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d6015dd70 msgr2=0x7f2d6006b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.306+0000 7f2d66963640 1 --2- 172.21.15.148:0/621078638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d6015dd70 0x7f2d6006b390 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2d500097a0 tx=0x7f2d5002a1e0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.306+0000 7f2d66963640 1 -- 172.21.15.148:0/621078638 shutdown_connections 2024-06-09T22:00:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.306+0000 7f2d66963640 1 --2- 172.21.15.148:0/621078638 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d6015dd70 0x7f2d6006b390 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.306+0000 7f2d66963640 1 -- 172.21.15.148:0/621078638 >> 172.21.15.148:0/621078638 conn(0x7f2d60070460 msgr2=0x7f2d600cc430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.306+0000 7f2d66963640 1 -- 172.21.15.148:0/621078638 shutdown_connections 2024-06-09T22:00:30.291 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.306+0000 7f2d66963640 1 -- 172.21.15.148:0/621078638 wait complete. 2024-06-09T22:00:30.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.307+0000 7f2d66963640 1 Processor -- start 2024-06-09T22:00:30.292 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.308+0000 7f2d66963640 1 -- start start 2024-06-09T22:00:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.308+0000 7f2d66963640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2d60073110 0x7f2d60148250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.308+0000 7f2d66963640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d600c7620 0x7f2d60151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.308+0000 7f2d66963640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d6015dd70 0x7f2d601e3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.308+0000 7f2d66963640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2d60072bf0 con 0x7f2d6015dd70 2024-06-09T22:00:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.308+0000 7f2d66963640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2d60072910 con 0x7f2d600c7620 2024-06-09T22:00:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.308+0000 7f2d66963640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f2d600cd220 con 0x7f2d60073110 2024-06-09T22:00:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.309+0000 7f2d5f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d600c7620 0x7f2d60151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.293 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.309+0000 7f2d5ffff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2d60073110 0x7f2d60148250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.309+0000 7f2d64f00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d6015dd70 0x7f2d601e3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.309+0000 7f2d64f00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d6015dd70 0x7f2d601e3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.148:48782/0 (socket says 172.21.15.148:48782) 2024-06-09T22:00:30.294 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.309+0000 7f2d64f00640 1 -- 172.21.15.148:0/3331978126 learned_addr learned my addr 172.21.15.148:0/3331978126 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.313+0000 7f2d64f00640 1 -- 172.21.15.148:0/3331978126 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2d60073110 msgr2=0x7f2d60148250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.313+0000 7f2d64f00640 1 --2- 172.21.15.148:0/3331978126 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2d60073110 0x7f2d60148250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.313+0000 7f2d64f00640 1 -- 172.21.15.148:0/3331978126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d600c7620 msgr2=0x7f2d60151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.313+0000 7f2d64f00640 1 --2- 172.21.15.148:0/3331978126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d600c7620 0x7f2d60151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.297 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.313+0000 7f2d64f00640 1 -- 172.21.15.148:0/3331978126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d601e5720 con 0x7f2d6015dd70 2024-06-09T22:00:30.299 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.315+0000 7f2d64f00640 1 --2- 172.21.15.148:0/3331978126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d6015dd70 0x7f2d601e3760 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800cbf0 tx=0x7f2d5800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:30.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.319+0000 7f2d5cff9640 1 -- 172.21.15.148:0/3331978126 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58017070 con 0x7f2d6015dd70 2024-06-09T22:00:30.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.319+0000 7f2d5cff9640 1 -- 172.21.15.148:0/3331978126 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d5800ad40 con 0x7f2d6015dd70 2024-06-09T22:00:30.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.319+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d601e5a00 con 0x7f2d6015dd70 2024-06-09T22:00:30.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.319+0000 7f2d5cff9640 1 -- 172.21.15.148:0/3331978126 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d58012600 con 0x7f2d6015dd70 2024-06-09T22:00:30.303 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.319+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d6015ce20 con 0x7f2d6015dd70 2024-06-09T22:00:30.304 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.320+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d6006b7d0 con 0x7f2d6015dd70 2024-06-09T22:00:30.305 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:30.322+0000 7fa707bb8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:00:30.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.323+0000 7f2d5cff9640 1 -- 172.21.15.148:0/3331978126 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d5801a070 con 0x7f2d6015dd70 2024-06-09T22:00:30.307 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.323+0000 7f2d5cff9640 1 -- 172.21.15.148:0/3331978126 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f2d5801c7a0 con 0x7f2d6015dd70 2024-06-09T22:00:30.355 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:30.371+0000 7fa707bb8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:00:30.404 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:30.420+0000 7f4ccb930040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-09T22:00:30.411 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:30.427+0000 7fa707bb8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:00:30.432 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.448+0000 7f2d5cff9640 1 -- 172.21.15.148:0/3331978126 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2d6006b7d0 con 0x7f2d6015dd70 2024-06-09T22:00:30.456 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:30.472+0000 7f4ccb930040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-09T22:00:30.504 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.520+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8db4805f-af3c-41a5-8eaf-1485c4ec8a5e", "id": 1} v 0) -- 0x7f2d601e5f20 con 0x7f2d6015dd70 2024-06-09T22:00:30.510 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:30.526+0000 7f4ccb930040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-09T22:00:30.530 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:30.546+0000 7fa707bb8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:00:30.549 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.565+0000 7f2d5cff9640 1 -- 172.21.15.148:0/3331978126 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8db4805f-af3c-41a5-8eaf-1485c4ec8a5e", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f2d5801c9f0 con 0x7f2d6015dd70 2024-06-09T22:00:30.550 INFO:teuthology.orchestra.run.smithi148.stdout:1 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.566+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d6015dd70 msgr2=0x7f2d601e3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.566+0000 7f2d66963640 1 --2- 172.21.15.148:0/3331978126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d6015dd70 0x7f2d601e3760 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2d5800cbf0 tx=0x7f2d5800a430 comp rx=0 tx=0).stop 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.566+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 shutdown_connections 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.566+0000 7f2d66963640 1 --2- 172.21.15.148:0/3331978126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d6015dd70 0x7f2d601e3760 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.566+0000 7f2d66963640 1 --2- 172.21.15.148:0/3331978126 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2d600c7620 0x7f2d60151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.566+0000 7f2d66963640 1 --2- 172.21.15.148:0/3331978126 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2d60073110 0x7f2d60148250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.566+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 >> 172.21.15.148:0/3331978126 conn(0x7f2d60070460 msgr2=0x7f2d6014ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.567+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 shutdown_connections 2024-06-09T22:00:30.551 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.567+0000 7f2d66963640 1 -- 172.21.15.148:0/3331978126 wait complete. 2024-06-09T22:00:30.562 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new 172c9719-1b63-460d-ba3e-dba05f194990 2 2024-06-09T22:00:30.631 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:30.647+0000 7f4ccb930040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-09T22:00:30.702 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.719+0000 7f7b70522640 1 Processor -- start 2024-06-09T22:00:30.703 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.719+0000 7f7b70522640 1 -- start start 2024-06-09T22:00:30.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.720+0000 7f7b70522640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68152390 0x7f7b680c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.720+0000 7f7b70522640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b68158d50 0x7f7b6815c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.720+0000 7f7b70522640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b680c8bf0 con 0x7f7b680c9b00 2024-06-09T22:00:30.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.720+0000 7f7b70522640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f7b68152770 con 0x7f7b68158d50 2024-06-09T22:00:30.704 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.720+0000 7f7b70522640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f7b6815b8c0 con 0x7f7b68152390 2024-06-09T22:00:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.720+0000 7f7b6dabd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b68158d50 0x7f7b6815c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.720+0000 7f7b6e2be640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68152390 0x7f7b680c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.721+0000 7f7b6dabd640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f7b68158d50 0x7f7b6815c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.148:48790/0 (socket says 172.21.15.148:48790) 2024-06-09T22:00:30.705 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.721+0000 7f7b6dabd640 1 -- 172.21.15.148:0/3315213674 learned_addr learned my addr 172.21.15.148:0/3315213674 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:30.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.723+0000 7f7b6e2be640 1 -- 172.21.15.148:0/3315213674 >> v2:172.21.15.32:3300/0 conn(0x7f7b68158d50 msgr2=0x7f7b6815c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:30.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.723+0000 7f7b6e2be640 1 --2- 172.21.15.148:0/3315213674 >> v2:172.21.15.32:3300/0 conn(0x7f7b68158d50 0x7f7b6815c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.723+0000 7f7b6e2be640 1 -- 172.21.15.148:0/3315213674 >> v1:172.21.15.32:6789/0 conn(0x7f7b680c9b00 legacy=0x7f7b680cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.707 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.723+0000 7f7b6e2be640 1 -- 172.21.15.148:0/3315213674 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b680c8ed0 con 0x7f7b68152390 2024-06-09T22:00:30.708 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.725+0000 7f7b6e2be640 1 --2- 172.21.15.148:0/3315213674 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68152390 0x7f7b680c2c70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7b640097a0 tx=0x7f7b6402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae8a938fdd9417da server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:30.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.728+0000 7f7b6d2bc640 1 -- 172.21.15.148:0/3315213674 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64037070 con 0x7f7b68152390 2024-06-09T22:00:30.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.728+0000 7f7b6d2bc640 1 -- 172.21.15.148:0/3315213674 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b64032440 con 0x7f7b68152390 2024-06-09T22:00:30.712 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.728+0000 7f7b6d2bc640 1 -- 172.21.15.148:0/3315213674 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b64032810 con 0x7f7b68152390 2024-06-09T22:00:30.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.729+0000 7f7b70522640 1 -- 172.21.15.148:0/3315213674 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68152390 msgr2=0x7f7b680c2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.729+0000 7f7b70522640 1 --2- 172.21.15.148:0/3315213674 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68152390 0x7f7b680c2c70 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f7b640097a0 tx=0x7f7b6402a210 comp rx=0 tx=0).stop 2024-06-09T22:00:30.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.729+0000 7f7b70522640 1 -- 172.21.15.148:0/3315213674 shutdown_connections 2024-06-09T22:00:30.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.729+0000 7f7b70522640 1 --2- 172.21.15.148:0/3315213674 >> v2:172.21.15.32:3300/0 conn(0x7f7b68158d50 0x7f7b6815c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.729+0000 7f7b70522640 1 --2- 172.21.15.148:0/3315213674 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68152390 0x7f7b680c2c70 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.713 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.729+0000 7f7b70522640 1 -- 172.21.15.148:0/3315213674 >> 172.21.15.148:0/3315213674 conn(0x7f7b680717f0 msgr2=0x7f7b68071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:30.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.729+0000 7f7b70522640 1 -- 172.21.15.148:0/3315213674 shutdown_connections 2024-06-09T22:00:30.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.729+0000 7f7b70522640 1 -- 172.21.15.148:0/3315213674 wait complete. 2024-06-09T22:00:30.714 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.730+0000 7f7b70522640 1 Processor -- start 2024-06-09T22:00:30.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.731+0000 7f7b70522640 1 -- start start 2024-06-09T22:00:30.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.731+0000 7f7b70522640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b680c9b00 0x7f7b680cc780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.731+0000 7f7b70522640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b68152390 0x7f7b680c7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.731+0000 7f7b70522640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68158d50 0x7f7b680c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:30.715 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.731+0000 7f7b70522640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7b680c92b0 con 0x7f7b68152390 2024-06-09T22:00:30.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.731+0000 7f7b70522640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f7b6815bdf0 con 0x7f7b680c9b00 2024-06-09T22:00:30.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.731+0000 7f7b70522640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f7b680ce2f0 con 0x7f7b68158d50 2024-06-09T22:00:30.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.732+0000 7f7b6eabf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68158d50 0x7f7b680c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.732+0000 7f7b6e2be640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b680c9b00 0x7f7b680cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.716 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.732+0000 7f7b6dabd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b68152390 0x7f7b680c7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:30.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.732+0000 7f7b6eabf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68158d50 0x7f7b680c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:44364/0 (socket says 172.21.15.148:44364) 2024-06-09T22:00:30.717 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.732+0000 7f7b6eabf640 1 -- 172.21.15.148:0/522498389 learned_addr learned my addr 172.21.15.148:0/522498389 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:30.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.736+0000 7f7b6dabd640 1 -- 172.21.15.148:0/522498389 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68158d50 msgr2=0x7f7b680c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.736+0000 7f7b6dabd640 1 --2- 172.21.15.148:0/522498389 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68158d50 0x7f7b680c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.736+0000 7f7b6dabd640 1 -- 172.21.15.148:0/522498389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b680c9b00 msgr2=0x7f7b680cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.736+0000 7f7b6dabd640 1 --2- 172.21.15.148:0/522498389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b680c9b00 0x7f7b680cc780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.720 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.736+0000 7f7b6dabd640 1 -- 172.21.15.148:0/522498389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b680cc530 con 0x7f7b68152390 2024-06-09T22:00:30.722 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.738+0000 7f7b6dabd640 1 --2- 172.21.15.148:0/522498389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b68152390 0x7f7b680c7210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800e880 tx=0x7f7b5800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:30.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.741+0000 7f7b5effd640 1 -- 172.21.15.148:0/522498389 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b58017070 con 0x7f7b68152390 2024-06-09T22:00:30.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.742+0000 7f7b5effd640 1 -- 172.21.15.148:0/522498389 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b5800c940 con 0x7f7b68152390 2024-06-09T22:00:30.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.742+0000 7f7b5effd640 1 -- 172.21.15.148:0/522498389 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b5800cd10 con 0x7f7b68152390 2024-06-09T22:00:30.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.742+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b680c57f0 con 0x7f7b68152390 2024-06-09T22:00:30.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.742+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b68159fb0 con 0x7f7b68152390 2024-06-09T22:00:30.726 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.742+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b68152900 con 0x7f7b68152390 2024-06-09T22:00:30.727 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.744+0000 7f7b5effd640 1 -- 172.21.15.148:0/522498389 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7b5801a070 con 0x7f7b68152390 2024-06-09T22:00:30.729 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.745+0000 7f7b5effd640 1 -- 172.21.15.148:0/522498389 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f7b58005990 con 0x7f7b68152390 2024-06-09T22:00:30.837 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.853+0000 7f7b5effd640 1 -- 172.21.15.148:0/522498389 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7b68152900 con 0x7f7b68152390 2024-06-09T22:00:30.907 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.923+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd new", "uuid": "172c9719-1b63-460d-ba3e-dba05f194990", "id": 2} v 0) -- 0x7f7b680c9920 con 0x7f7b68152390 2024-06-09T22:00:30.938 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:30.954+0000 7fa707bb8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:00:30.939 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.955+0000 7f7b5effd640 1 -- 172.21.15.148:0/522498389 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "172c9719-1b63-460d-ba3e-dba05f194990", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f7b680c9920 con 0x7f7b68152390 2024-06-09T22:00:30.939 INFO:teuthology.orchestra.run.smithi148.stdout:2 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.956+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b68152390 msgr2=0x7f7b680c7210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.956+0000 7f7b70522640 1 --2- 172.21.15.148:0/522498389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b68152390 0x7f7b680c7210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7b5800e880 tx=0x7f7b5800ed40 comp rx=0 tx=0).stop 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.956+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 shutdown_connections 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.956+0000 7f7b70522640 1 --2- 172.21.15.148:0/522498389 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f7b68158d50 0x7f7b680c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.956+0000 7f7b70522640 1 --2- 172.21.15.148:0/522498389 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b68152390 0x7f7b680c7210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.956+0000 7f7b70522640 1 --2- 172.21.15.148:0/522498389 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f7b680c9b00 0x7f7b680cc780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.956+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 >> 172.21.15.148:0/522498389 conn(0x7f7b680717f0 msgr2=0x7f7b680ca2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.957+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 shutdown_connections 2024-06-09T22:00:30.941 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:30.957+0000 7f7b70522640 1 -- 172.21.15.148:0/522498389 wait complete. 2024-06-09T22:00:30.951 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new 611e5921-1524-4cc9-9f33-a51d431115f2 3 2024-06-09T22:00:31.025 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:31.041+0000 7f4ccb930040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-09T22:00:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.108+0000 7f074fe1c640 1 Processor -- start 2024-06-09T22:00:31.092 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.108+0000 7f074fe1c640 1 -- start start 2024-06-09T22:00:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.109+0000 7f074fe1c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07480c7be0 0x7f074815e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.109+0000 7f074fe1c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f074806f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.109+0000 7f074fe1c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f074806f000 con 0x7f0748154850 2024-06-09T22:00:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.109+0000 7f074fe1c640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f074806fe70 con 0x7f07480c7be0 2024-06-09T22:00:31.093 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.109+0000 7f074fe1c640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f074815e310 con 0x7f07480703a0 2024-06-09T22:00:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.109+0000 7f074e3b9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f074806f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.110+0000 7f074d3b7640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f07480c7be0 0x7f074815e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.110+0000 7f074e3b9640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f074806f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:48026/0 (socket says 172.21.15.148:48026) 2024-06-09T22:00:31.094 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.110+0000 7f074e3b9640 1 -- 172.21.15.148:0/966092957 learned_addr learned my addr 172.21.15.148:0/966092957 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.112+0000 7f074e3b9640 1 -- 172.21.15.148:0/966092957 >> v2:172.21.15.32:3300/0 conn(0x7f07480c7be0 msgr2=0x7f074815e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.112+0000 7f074e3b9640 1 --2- 172.21.15.148:0/966092957 >> v2:172.21.15.32:3300/0 conn(0x7f07480c7be0 0x7f074815e7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.112+0000 7f074e3b9640 1 -- 172.21.15.148:0/966092957 >> v1:172.21.15.32:6789/0 conn(0x7f0748154850 legacy=0x7f074806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.096 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.112+0000 7f074e3b9640 1 -- 172.21.15.148:0/966092957 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f074806ea40 con 0x7f07480703a0 2024-06-09T22:00:31.098 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.114+0000 7f074e3b9640 1 --2- 172.21.15.148:0/966092957 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f074806f930 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0740009e50 tx=0x7f074002a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7825340a44642e05 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.117+0000 7f074cbb6640 1 -- 172.21.15.148:0/966092957 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0740036070 con 0x7f07480703a0 2024-06-09T22:00:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.117+0000 7f074cbb6640 1 -- 172.21.15.148:0/966092957 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0740031600 con 0x7f07480703a0 2024-06-09T22:00:31.101 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.117+0000 7f074cbb6640 1 -- 172.21.15.148:0/966092957 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07400319d0 con 0x7f07480703a0 2024-06-09T22:00:31.102 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.118+0000 7f074fe1c640 1 -- 172.21.15.148:0/966092957 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 msgr2=0x7f074806f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.118+0000 7f074fe1c640 1 --2- 172.21.15.148:0/966092957 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f074806f930 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0740009e50 tx=0x7f074002a7a0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.118+0000 7f074fe1c640 1 -- 172.21.15.148:0/966092957 shutdown_connections 2024-06-09T22:00:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.118+0000 7f074fe1c640 1 --2- 172.21.15.148:0/966092957 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f074806f930 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.118+0000 7f074fe1c640 1 --2- 172.21.15.148:0/966092957 >> v2:172.21.15.32:3300/0 conn(0x7f07480c7be0 0x7f074815e7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.118+0000 7f074fe1c640 1 -- 172.21.15.148:0/966092957 >> 172.21.15.148:0/966092957 conn(0x7f07480cce30 msgr2=0x7f07480cd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.118+0000 7f074fe1c640 1 -- 172.21.15.148:0/966092957 shutdown_connections 2024-06-09T22:00:31.103 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.118+0000 7f074fe1c640 1 -- 172.21.15.148:0/966092957 wait complete. 2024-06-09T22:00:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.120+0000 7f074fe1c640 1 Processor -- start 2024-06-09T22:00:31.104 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.120+0000 7f074fe1c640 1 -- start start 2024-06-09T22:00:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.121+0000 7f074fe1c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f07481e6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.121+0000 7f074fe1c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07480c7be0 0x7f07480c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.106 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.121+0000 7f074fe1c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0748154850 0x7f07480c4230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.121+0000 7f074fe1c640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f074815dcc0 con 0x7f07480c7be0 2024-06-09T22:00:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.121+0000 7f074fe1c640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0748070850 con 0x7f0748154850 2024-06-09T22:00:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.121+0000 7f074fe1c640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f074815dfa0 con 0x7f07480703a0 2024-06-09T22:00:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.122+0000 7f074dbb8640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f07481e6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.122+0000 7f074e3b9640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0748154850 0x7f07480c4230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.122+0000 7f074d3b7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07480c7be0 0x7f07480c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.122+0000 7f074dbb8640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f07481e6120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:48034/0 (socket says 172.21.15.148:48034) 2024-06-09T22:00:31.107 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.122+0000 7f074dbb8640 1 -- 172.21.15.148:0/1525266484 learned_addr learned my addr 172.21.15.148:0/1525266484 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.126+0000 7f074d3b7640 1 -- 172.21.15.148:0/1525266484 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 msgr2=0x7f07481e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.126+0000 7f074d3b7640 1 --2- 172.21.15.148:0/1525266484 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f07481e6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.126+0000 7f074d3b7640 1 -- 172.21.15.148:0/1525266484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0748154850 msgr2=0x7f07480c4230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.126+0000 7f074d3b7640 1 --2- 172.21.15.148:0/1525266484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0748154850 0x7f07480c4230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.110 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.126+0000 7f074d3b7640 1 -- 172.21.15.148:0/1525266484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07480c61b0 con 0x7f07480c7be0 2024-06-09T22:00:31.112 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.128+0000 7f074d3b7640 1 --2- 172.21.15.148:0/1525266484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07480c7be0 0x7f07480c69e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f073800e850 tx=0x7f073800ed10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.132+0000 7f073e7fc640 1 -- 172.21.15.148:0/1525266484 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0738017070 con 0x7f07480c7be0 2024-06-09T22:00:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.132+0000 7f073e7fc640 1 -- 172.21.15.148:0/1525266484 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f073800c940 con 0x7f07480c7be0 2024-06-09T22:00:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.132+0000 7f073e7fc640 1 -- 172.21.15.148:0/1525266484 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f073800cd10 con 0x7f07480c7be0 2024-06-09T22:00:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.132+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07481e4630 con 0x7f07480c7be0 2024-06-09T22:00:31.116 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.132+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07481e4bf0 con 0x7f07480c7be0 2024-06-09T22:00:31.117 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.133+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0748075af0 con 0x7f07480c7be0 2024-06-09T22:00:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.135+0000 7f073e7fc640 1 -- 172.21.15.148:0/1525266484 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f073801a070 con 0x7f07480c7be0 2024-06-09T22:00:31.119 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.135+0000 7f073e7fc640 1 -- 172.21.15.148:0/1525266484 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f0738013470 con 0x7f07480c7be0 2024-06-09T22:00:31.214 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:31.230+0000 7fa707bb8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:00:31.228 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.244+0000 7f073e7fc640 1 -- 172.21.15.148:0/1525266484 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f0748075af0 con 0x7f07480c7be0 2024-06-09T22:00:31.272 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:31.288+0000 7fa707bb8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:00:31.333 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:31.316+0000 7f4ccb930040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-09T22:00:31.333 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.322+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd new", "uuid": "611e5921-1524-4cc9-9f33-a51d431115f2", "id": 3} v 0) -- 0x7f0748074f90 con 0x7f07480c7be0 2024-06-09T22:00:31.354 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:31.370+0000 7f4ccb930040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-09T22:00:31.357 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.373+0000 7f073e7fc640 1 -- 172.21.15.148:0/1525266484 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "611e5921-1524-4cc9-9f33-a51d431115f2", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f07380065b0 con 0x7f07480c7be0 2024-06-09T22:00:31.357 INFO:teuthology.orchestra.run.smithi148.stdout:3 2024-06-09T22:00:31.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07480c7be0 msgr2=0x7f07480c69e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 --2- 172.21.15.148:0/1525266484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07480c7be0 0x7f07480c69e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f073800e850 tx=0x7f073800ed10 comp rx=0 tx=0).stop 2024-06-09T22:00:31.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 shutdown_connections 2024-06-09T22:00:31.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 --2- 172.21.15.148:0/1525266484 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0748154850 0x7f07480c4230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 --2- 172.21.15.148:0/1525266484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07480c7be0 0x7f07480c69e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.358 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 --2- 172.21.15.148:0/1525266484 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f07480703a0 0x7f07481e6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 >> 172.21.15.148:0/1525266484 conn(0x7f07480cce30 msgr2=0x7f074815c2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:31.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 shutdown_connections 2024-06-09T22:00:31.359 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.374+0000 7f074fe1c640 1 -- 172.21.15.148:0/1525266484 wait complete. 2024-06-09T22:00:31.369 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new b85f7437-19ea-4f08-8a71-ef14330e0351 4 2024-06-09T22:00:31.399 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:31.415+0000 7fa707bb8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:00:31.480 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:31.496+0000 7f4ccb930040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-09T22:00:31.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.526+0000 7ff158aba640 1 Processor -- start 2024-06-09T22:00:31.510 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.526+0000 7ff158aba640 1 -- start start 2024-06-09T22:00:31.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.527+0000 7ff158aba640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff1540cdde0 0x7ff154070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.527+0000 7ff158aba640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff1540c8770 0x7ff15415bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.527+0000 7ff158aba640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff154158450 con 0x7ff1540ce1c0 2024-06-09T22:00:31.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.527+0000 7ff158aba640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7ff15415b7b0 con 0x7ff1540c8770 2024-06-09T22:00:31.511 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.527+0000 7ff158aba640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7ff154158730 con 0x7ff1540cdde0 2024-06-09T22:00:31.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.527+0000 7ff15259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff1540cdde0 0x7ff154070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.528+0000 7ff15259c640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7ff1540cdde0 0x7ff154070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42828/0 (socket says 172.21.15.148:42828) 2024-06-09T22:00:31.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.528+0000 7ff15259c640 1 -- 172.21.15.148:0/4103636640 learned_addr learned my addr 172.21.15.148:0/4103636640 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:31.512 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.528+0000 7ff152d9d640 1 --2- 172.21.15.148:0/4103636640 >> v2:172.21.15.32:3300/0 conn(0x7ff1540c8770 0x7ff15415bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.530+0000 7ff15259c640 1 -- 172.21.15.148:0/4103636640 >> v2:172.21.15.32:3300/0 conn(0x7ff1540c8770 msgr2=0x7ff15415bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:31.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.530+0000 7ff15259c640 1 --2- 172.21.15.148:0/4103636640 >> v2:172.21.15.32:3300/0 conn(0x7ff1540c8770 0x7ff15415bc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.514 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.530+0000 7ff15259c640 1 -- 172.21.15.148:0/4103636640 >> v1:172.21.15.148:6789/0 conn(0x7ff1540ce1c0 legacy=0x7ff154152110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:31.515 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.530+0000 7ff15259c640 1 -- 172.21.15.148:0/4103636640 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff154158a10 con 0x7ff1540cdde0 2024-06-09T22:00:31.516 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.532+0000 7ff15259c640 1 --2- 172.21.15.148:0/4103636640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540cdde0 0x7ff154070ac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff154072160 tx=0x7ff14802a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6650923cce58d39 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:31.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.536+0000 7ff15159a640 1 -- 172.21.15.148:0/4103636640 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148037070 con 0x7ff1540cdde0 2024-06-09T22:00:31.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.536+0000 7ff15159a640 1 -- 172.21.15.148:0/4103636640 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff148032440 con 0x7ff1540cdde0 2024-06-09T22:00:31.520 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.536+0000 7ff15159a640 1 -- 172.21.15.148:0/4103636640 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff148032810 con 0x7ff1540cdde0 2024-06-09T22:00:31.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.537+0000 7ff158aba640 1 -- 172.21.15.148:0/4103636640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540cdde0 msgr2=0x7ff154070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.537+0000 7ff158aba640 1 --2- 172.21.15.148:0/4103636640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540cdde0 0x7ff154070ac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff154072160 tx=0x7ff14802a2c0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.537+0000 7ff158aba640 1 -- 172.21.15.148:0/4103636640 shutdown_connections 2024-06-09T22:00:31.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.537+0000 7ff158aba640 1 --2- 172.21.15.148:0/4103636640 >> v2:172.21.15.32:3300/0 conn(0x7ff1540c8770 0x7ff15415bc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.537+0000 7ff158aba640 1 --2- 172.21.15.148:0/4103636640 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540cdde0 0x7ff154070ac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.521 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.537+0000 7ff158aba640 1 -- 172.21.15.148:0/4103636640 >> 172.21.15.148:0/4103636640 conn(0x7ff154062520 msgr2=0x7ff154071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:31.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.537+0000 7ff158aba640 1 -- 172.21.15.148:0/4103636640 shutdown_connections 2024-06-09T22:00:31.522 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.537+0000 7ff158aba640 1 -- 172.21.15.148:0/4103636640 wait complete. 2024-06-09T22:00:31.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.539+0000 7ff158aba640 1 Processor -- start 2024-06-09T22:00:31.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.539+0000 7ff158aba640 1 -- start start 2024-06-09T22:00:31.523 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.539+0000 7ff158aba640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1540c8770 0x7ff154153940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.539+0000 7ff158aba640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff1540cdde0 0x7ff1540c6010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.539+0000 7ff158aba640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ce1c0 0x7ff1540c7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:31.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.539+0000 7ff158aba640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff154152820 con 0x7ff1540c8770 2024-06-09T22:00:31.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.539+0000 7ff158aba640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff154158fd0 con 0x7ff1540ce1c0 2024-06-09T22:00:31.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.539+0000 7ff158aba640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7ff1541529a0 con 0x7ff1540cdde0 2024-06-09T22:00:31.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.540+0000 7ff152d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ce1c0 0x7ff1540c7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.524 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.540+0000 7ff151d9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff1540cdde0 0x7ff1540c6010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.540+0000 7ff152d9d640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ce1c0 0x7ff1540c7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42842/0 (socket says 172.21.15.148:42842) 2024-06-09T22:00:31.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.540+0000 7ff152d9d640 1 -- 172.21.15.148:0/2682864910 learned_addr learned my addr 172.21.15.148:0/2682864910 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:31.525 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.540+0000 7ff15259c640 1 --2- 172.21.15.148:0/2682864910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1540c8770 0x7ff154153940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:31.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.544+0000 7ff152d9d640 1 -- 172.21.15.148:0/2682864910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff1540cdde0 msgr2=0x7ff1540c6010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.544+0000 7ff152d9d640 1 --2- 172.21.15.148:0/2682864910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff1540cdde0 0x7ff1540c6010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.544+0000 7ff152d9d640 1 -- 172.21.15.148:0/2682864910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1540c8770 msgr2=0x7ff154153940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.544+0000 7ff152d9d640 1 --2- 172.21.15.148:0/2682864910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1540c8770 0x7ff154153940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.528 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.544+0000 7ff152d9d640 1 -- 172.21.15.148:0/2682864910 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff15414e1b0 con 0x7ff1540ce1c0 2024-06-09T22:00:31.529 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.546+0000 7ff152d9d640 1 --2- 172.21.15.148:0/2682864910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ce1c0 0x7ff1540c7130 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00efc0 tx=0x7ff14c00c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:31.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.549+0000 7ff142ffd640 1 -- 172.21.15.148:0/2682864910 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14c018020 con 0x7ff1540ce1c0 2024-06-09T22:00:31.533 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.549+0000 7ff142ffd640 1 -- 172.21.15.148:0/2682864910 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff14c014480 con 0x7ff1540ce1c0 2024-06-09T22:00:31.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.549+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1541f16d0 con 0x7ff1540ce1c0 2024-06-09T22:00:31.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.549+0000 7ff142ffd640 1 -- 172.21.15.148:0/2682864910 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff14c014870 con 0x7ff1540ce1c0 2024-06-09T22:00:31.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.549+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1541f1c90 con 0x7ff1540ce1c0 2024-06-09T22:00:31.534 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.550+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff154068490 con 0x7ff1540ce1c0 2024-06-09T22:00:31.710 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:31.727+0000 7fa707bb8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:00:31.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.734+0000 7ff142ffd640 1 -- 172.21.15.148:0/2682864910 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff14c005d10 con 0x7ff1540ce1c0 2024-06-09T22:00:31.718 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.734+0000 7ff142ffd640 1 -- 172.21.15.148:0/2682864910 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7ff14c0138f0 con 0x7ff1540ce1c0 2024-06-09T22:00:31.719 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.734+0000 7ff142ffd640 1 -- 172.21.15.148:0/2682864910 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7ff14c022030 con 0x7ff1540ce1c0 2024-06-09T22:00:31.760 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:31.777+0000 7fa707bb8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:00:31.783 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:31.799+0000 7f4ccb930040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-09T22:00:31.791 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.807+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b85f7437-19ea-4f08-8a71-ef14330e0351", "id": 4} v 0) -- 0x7ff15414e740 con 0x7ff1540ce1c0 2024-06-09T22:00:31.832 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:31.848+0000 7f4ccb930040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-09T22:00:31.918 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.934+0000 7ff142ffd640 1 -- 172.21.15.148:0/2682864910 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b85f7437-19ea-4f08-8a71-ef14330e0351", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7ff14c010040 con 0x7ff1540ce1c0 2024-06-09T22:00:31.918 INFO:teuthology.orchestra.run.smithi148.stdout:4 2024-06-09T22:00:31.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ce1c0 msgr2=0x7ff1540c7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:31.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 --2- 172.21.15.148:0/2682864910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ce1c0 0x7ff1540c7130 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7ff14c00efc0 tx=0x7ff14c00c490 comp rx=0 tx=0).stop 2024-06-09T22:00:31.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 shutdown_connections 2024-06-09T22:00:31.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 --2- 172.21.15.148:0/2682864910 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff1540ce1c0 0x7ff1540c7130 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 --2- 172.21.15.148:0/2682864910 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff1540cdde0 0x7ff1540c6010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 --2- 172.21.15.148:0/2682864910 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1540c8770 0x7ff154153940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:31.919 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 >> 172.21.15.148:0/2682864910 conn(0x7ff154062520 msgr2=0x7ff1540c8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:31.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 shutdown_connections 2024-06-09T22:00:31.920 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:31.935+0000 7ff158aba640 1 -- 172.21.15.148:0/2682864910 wait complete. 2024-06-09T22:00:31.929 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:31.945+0000 7fa707bb8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:00:31.930 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new ae110414-eb0e-4253-91f2-0b2e83dad115 5 2024-06-09T22:00:31.999 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:32.015+0000 7f4ccb930040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-09T22:00:32.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.046+0000 7f0364514640 1 Processor -- start 2024-06-09T22:00:32.030 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.046+0000 7f0364514640 1 -- start start 2024-06-09T22:00:32.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.047+0000 7f0364514640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c158d50 0x7f035c0c2c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.047+0000 7f0364514640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f035c0c8bf0 con 0x7f035c0c9b00 2024-06-09T22:00:32.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.047+0000 7f0364514640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f035c0c8ed0 con 0x7f035c152390 2024-06-09T22:00:32.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.047+0000 7f0364514640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f035c152770 con 0x7f035c158d50 2024-06-09T22:00:32.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.047+0000 7f0361aaf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c158d50 0x7f035c0c2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.031 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.047+0000 7f03622b0640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f035c152390 0x7f035c0cdbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44586/0 (socket says 172.21.15.148:44586) 2024-06-09T22:00:32.032 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.047+0000 7f03622b0640 1 -- 172.21.15.148:0/1091424239 learned_addr learned my addr 172.21.15.148:0/1091424239 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-06-09T22:00:32.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.050+0000 7f03612ae640 1 -- 172.21.15.148:0/1091424239 <== mon.1 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2118422329 0 0) 0x7f035c0c8ed0 con 0x7f035c152390 2024-06-09T22:00:32.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.050+0000 7f03612ae640 1 -- 172.21.15.148:0/1091424239 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0348003520 con 0x7f035c152390 2024-06-09T22:00:32.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.050+0000 7f0361aaf640 1 -- 172.21.15.148:0/1091424239 >> v1:172.21.15.148:6789/0 conn(0x7f035c152390 legacy=0x7f035c0cdbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.050+0000 7f0361aaf640 1 -- 172.21.15.148:0/1091424239 >> v1:172.21.15.32:6789/0 conn(0x7f035c0c9b00 legacy=0x7f035c162c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.034 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.050+0000 7f0361aaf640 1 -- 172.21.15.148:0/1091424239 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035c15bb10 con 0x7f035c158d50 2024-06-09T22:00:32.035 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.052+0000 7f0361aaf640 1 --2- 172.21.15.148:0/1091424239 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c158d50 0x7f035c0c2c70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f034c0097d0 tx=0x7f034c02a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a0b4c640f5d0656 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:32.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.055+0000 7f03612ae640 1 -- 172.21.15.148:0/1091424239 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c037070 con 0x7f035c158d50 2024-06-09T22:00:32.039 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.055+0000 7f03612ae640 1 -- 172.21.15.148:0/1091424239 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034c032440 con 0x7f035c158d50 2024-06-09T22:00:32.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.055+0000 7f03612ae640 1 -- 172.21.15.148:0/1091424239 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c0327f0 con 0x7f035c158d50 2024-06-09T22:00:32.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.056+0000 7f0364514640 1 -- 172.21.15.148:0/1091424239 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c158d50 msgr2=0x7f035c0c2c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.056+0000 7f0364514640 1 --2- 172.21.15.148:0/1091424239 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c158d50 0x7f035c0c2c70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f034c0097d0 tx=0x7f034c02a290 comp rx=0 tx=0).stop 2024-06-09T22:00:32.040 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.056+0000 7f0364514640 1 -- 172.21.15.148:0/1091424239 shutdown_connections 2024-06-09T22:00:32.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.056+0000 7f0364514640 1 --2- 172.21.15.148:0/1091424239 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c158d50 0x7f035c0c2c70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.056+0000 7f0364514640 1 -- 172.21.15.148:0/1091424239 >> 172.21.15.148:0/1091424239 conn(0x7f035c0717f0 msgr2=0x7f035c071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:32.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.056+0000 7f0364514640 1 -- 172.21.15.148:0/1091424239 shutdown_connections 2024-06-09T22:00:32.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.056+0000 7f0364514640 1 -- 172.21.15.148:0/1091424239 wait complete. 2024-06-09T22:00:32.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.058+0000 7f0364514640 1 Processor -- start 2024-06-09T22:00:32.041 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.058+0000 7f0364514640 1 -- start start 2024-06-09T22:00:32.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.058+0000 7f0364514640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0c9b00 0x7f035c1f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.042 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.058+0000 7f0364514640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c152390 0x7f035c1e3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.058+0000 7f0364514640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c158d50 0x7f035c1e2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.058+0000 7f0364514640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f035c0c92b0 con 0x7f035c158d50 2024-06-09T22:00:32.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.058+0000 7f0364514640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f035c15b8c0 con 0x7f035c0c9b00 2024-06-09T22:00:32.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.058+0000 7f0364514640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f035c0ce2f0 con 0x7f035c152390 2024-06-09T22:00:32.043 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.059+0000 7f0361aaf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c152390 0x7f035c1e3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.059+0000 7f03622b0640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0c9b00 0x7f035c1f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.059+0000 7f0362ab1640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c158d50 0x7f035c1e2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.059+0000 7f0361aaf640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c152390 0x7f035c1e3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:48060/0 (socket says 172.21.15.148:48060) 2024-06-09T22:00:32.044 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.059+0000 7f0361aaf640 1 -- 172.21.15.148:0/536555898 learned_addr learned my addr 172.21.15.148:0/536555898 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:32.046 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.063+0000 7f0361aaf640 1 -- 172.21.15.148:0/536555898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0c9b00 msgr2=0x7f035c1f0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.063+0000 7f0361aaf640 1 --2- 172.21.15.148:0/536555898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0c9b00 0x7f035c1f0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.063+0000 7f0361aaf640 1 -- 172.21.15.148:0/536555898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c158d50 msgr2=0x7f035c1e2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.063+0000 7f0361aaf640 1 --2- 172.21.15.148:0/536555898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c158d50 0x7f035c1e2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.047 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.063+0000 7f0361aaf640 1 -- 172.21.15.148:0/536555898 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035c0cc2a0 con 0x7f035c152390 2024-06-09T22:00:32.049 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.065+0000 7f0361aaf640 1 --2- 172.21.15.148:0/536555898 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c152390 0x7f035c1e3d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f034c0098f0 tx=0x7f034c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:32.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.068+0000 7f0352ffd640 1 -- 172.21.15.148:0/536555898 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c037040 con 0x7f035c152390 2024-06-09T22:00:32.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.069+0000 7f0352ffd640 1 -- 172.21.15.148:0/536555898 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034c042070 con 0x7f035c152390 2024-06-09T22:00:32.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.069+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f035c0cbce0 con 0x7f035c152390 2024-06-09T22:00:32.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.069+0000 7f0352ffd640 1 -- 172.21.15.148:0/536555898 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034c030940 con 0x7f035c152390 2024-06-09T22:00:32.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.069+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f035c1f02c0 con 0x7f035c152390 2024-06-09T22:00:32.053 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.070+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f035c074f90 con 0x7f035c152390 2024-06-09T22:00:32.077 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:32.093+0000 7fa707bb8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:00:32.129 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:32.145+0000 7fa707bb8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:00:32.146 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:32.163+0000 7f4ccb930040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-09T22:00:32.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.178+0000 7f0352ffd640 1 -- 172.21.15.148:0/536555898 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f034c030ae0 con 0x7f035c152390 2024-06-09T22:00:32.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.178+0000 7f0352ffd640 1 -- 172.21.15.148:0/536555898 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f034c047bc0 con 0x7f035c152390 2024-06-09T22:00:32.162 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.178+0000 7f0352ffd640 1 -- 172.21.15.148:0/536555898 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f034c051430 con 0x7f035c152390 2024-06-09T22:00:32.198 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:32.214+0000 7f4ccb930040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-09T22:00:32.203 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:32.219+0000 7fa707bb8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:00:32.240 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.256+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd new", "uuid": "ae110414-eb0e-4253-91f2-0b2e83dad115", "id": 5} v 0) -- 0x7f035c0c9590 con 0x7f035c152390 2024-06-09T22:00:32.252 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:32.268+0000 7fa707bb8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:00:32.275 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:32.292+0000 7f4ccb930040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-09T22:00:32.326 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:32.342+0000 7f4ccb930040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-09T22:00:32.365 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.381+0000 7f0352ffd640 1 -- 172.21.15.148:0/536555898 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae110414-eb0e-4253-91f2-0b2e83dad115", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f034c047460 con 0x7f035c152390 2024-06-09T22:00:32.366 INFO:teuthology.orchestra.run.smithi148.stdout:5 2024-06-09T22:00:32.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c152390 msgr2=0x7f035c1e3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.367 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 --2- 172.21.15.148:0/536555898 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c152390 0x7f035c1e3d50 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f034c0098f0 tx=0x7f034c0046c0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 shutdown_connections 2024-06-09T22:00:32.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 --2- 172.21.15.148:0/536555898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f035c158d50 0x7f035c1e2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 --2- 172.21.15.148:0/536555898 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f035c152390 0x7f035c1e3d50 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 --2- 172.21.15.148:0/536555898 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f035c0c9b00 0x7f035c1f0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 >> 172.21.15.148:0/536555898 conn(0x7f035c0717f0 msgr2=0x7f035c0b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:32.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 shutdown_connections 2024-06-09T22:00:32.368 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.383+0000 7f0364514640 1 -- 172.21.15.148:0/536555898 wait complete. 2024-06-09T22:00:32.381 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new 0de55946-8987-4376-b9f9-f4957200232f 6 2024-06-09T22:00:32.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.497+0000 7fe46cd6a640 1 Processor -- start 2024-06-09T22:00:32.481 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.497+0000 7fe46cd6a640 1 -- start start 2024-06-09T22:00:32.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.498+0000 7fe46cd6a640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe468151460 0x7fe4681587d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.498+0000 7fe46cd6a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe46806cdc0 0x7fe4680cca50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.498+0000 7fe46cd6a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe468158d10 con 0x7fe4680c4c10 2024-06-09T22:00:32.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.498+0000 7fe46cd6a640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7fe4680c3ee0 con 0x7fe468151460 2024-06-09T22:00:32.482 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.498+0000 7fe46cd6a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7fe4680c4780 con 0x7fe46806cdc0 2024-06-09T22:00:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.498+0000 7fe465d9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe46806cdc0 0x7fe4680cca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.499+0000 7fe46659c640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe468151460 0x7fe4681587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.499+0000 7fe465d9b640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe46806cdc0 0x7fe4680cca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.148:42864/0 (socket says 172.21.15.148:42864) 2024-06-09T22:00:32.483 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.499+0000 7fe465d9b640 1 -- 172.21.15.148:0/3678468528 learned_addr learned my addr 172.21.15.148:0/3678468528 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:32.486 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.502+0000 7fe46659c640 1 -- 172.21.15.148:0/3678468528 >> v2:172.21.15.148:3300/0 conn(0x7fe46806cdc0 msgr2=0x7fe4680cca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:32.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.502+0000 7fe46659c640 1 --2- 172.21.15.148:0/3678468528 >> v2:172.21.15.148:3300/0 conn(0x7fe46806cdc0 0x7fe4680cca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.502+0000 7fe46659c640 1 -- 172.21.15.148:0/3678468528 >> v1:172.21.15.32:6789/0 conn(0x7fe4680c4c10 legacy=0x7fe4681572e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.487 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.503+0000 7fe46659c640 1 -- 172.21.15.148:0/3678468528 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe468157e70 con 0x7fe468151460 2024-06-09T22:00:32.489 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.505+0000 7fe46659c640 1 --2- 172.21.15.148:0/3678468528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 0x7fe4681587d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe4680c4060 tx=0x7fe45802a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bac3ea2bbd79017c server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:32.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46559a640 1 -- 172.21.15.148:0/3678468528 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458037070 con 0x7fe468151460 2024-06-09T22:00:32.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46559a640 1 -- 172.21.15.148:0/3678468528 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe458032440 con 0x7fe468151460 2024-06-09T22:00:32.493 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46559a640 1 -- 172.21.15.148:0/3678468528 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458032810 con 0x7fe468151460 2024-06-09T22:00:32.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46cd6a640 1 -- 172.21.15.148:0/3678468528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 msgr2=0x7fe4681587d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46cd6a640 1 --2- 172.21.15.148:0/3678468528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 0x7fe4681587d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe4680c4060 tx=0x7fe45802a1a0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46cd6a640 1 -- 172.21.15.148:0/3678468528 shutdown_connections 2024-06-09T22:00:32.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46cd6a640 1 --2- 172.21.15.148:0/3678468528 >> v2:172.21.15.148:3300/0 conn(0x7fe46806cdc0 0x7fe4680cca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46cd6a640 1 --2- 172.21.15.148:0/3678468528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 0x7fe4681587d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46cd6a640 1 -- 172.21.15.148:0/3678468528 >> 172.21.15.148:0/3678468528 conn(0x7fe468072ad0 msgr2=0x7fe468072eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:32.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46cd6a640 1 -- 172.21.15.148:0/3678468528 shutdown_connections 2024-06-09T22:00:32.494 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.509+0000 7fe46cd6a640 1 -- 172.21.15.148:0/3678468528 wait complete. 2024-06-09T22:00:32.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.511+0000 7fe46cd6a640 1 Processor -- start 2024-06-09T22:00:32.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.511+0000 7fe46cd6a640 1 -- start start 2024-06-09T22:00:32.495 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe46cd6a640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe46806cdc0 0x7fe4681525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe46cd6a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4680c4c10 0x7fe46814d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe46cd6a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 0x7fe468153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe46cd6a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe4680c4410 con 0x7fe468151460 2024-06-09T22:00:32.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe46cd6a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe4680c4130 con 0x7fe4680c4c10 2024-06-09T22:00:32.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe46cd6a640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fe468151840 con 0x7fe46806cdc0 2024-06-09T22:00:32.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe46659c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe46806cdc0 0x7fe4681525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.496 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe465d9b640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4680c4c10 0x7fe46814d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.512+0000 7fe466d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 0x7fe468153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.513+0000 7fe46659c640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe46806cdc0 0x7fe4681525e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.148:48066/0 (socket says 172.21.15.148:48066) 2024-06-09T22:00:32.497 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.513+0000 7fe46659c640 1 -- 172.21.15.148:0/2183805422 learned_addr learned my addr 172.21.15.148:0/2183805422 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:32.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.516+0000 7fe46659c640 1 -- 172.21.15.148:0/2183805422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4680c4c10 msgr2=0x7fe46814d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.516+0000 7fe46659c640 1 --2- 172.21.15.148:0/2183805422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4680c4c10 0x7fe46814d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.516+0000 7fe46659c640 1 -- 172.21.15.148:0/2183805422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 msgr2=0x7fe468153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.516+0000 7fe46659c640 1 --2- 172.21.15.148:0/2183805422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 0x7fe468153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.500 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.516+0000 7fe46659c640 1 -- 172.21.15.148:0/2183805422 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4681549a0 con 0x7fe46806cdc0 2024-06-09T22:00:32.502 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.518+0000 7fe46659c640 1 --2- 172.21.15.148:0/2183805422 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe46806cdc0 0x7fe4681525e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe458009860 tx=0x7fe4580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:32.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.522+0000 7fe456ffd640 1 -- 172.21.15.148:0/2183805422 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458037040 con 0x7fe46806cdc0 2024-06-09T22:00:32.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.522+0000 7fe456ffd640 1 -- 172.21.15.148:0/2183805422 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe458042070 con 0x7fe46806cdc0 2024-06-09T22:00:32.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.522+0000 7fe456ffd640 1 -- 172.21.15.148:0/2183805422 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe458030960 con 0x7fe46806cdc0 2024-06-09T22:00:32.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.522+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4681f1b00 con 0x7fe46806cdc0 2024-06-09T22:00:32.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.522+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4681f20c0 con 0x7fe46806cdc0 2024-06-09T22:00:32.507 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.523+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe468074f90 con 0x7fe46806cdc0 2024-06-09T22:00:32.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.639+0000 7fe456ffd640 1 -- 172.21.15.148:0/2183805422 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe458030b00 con 0x7fe46806cdc0 2024-06-09T22:00:32.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.639+0000 7fe456ffd640 1 -- 172.21.15.148:0/2183805422 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fe458047b30 con 0x7fe46806cdc0 2024-06-09T22:00:32.624 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.639+0000 7fe456ffd640 1 -- 172.21.15.148:0/2183805422 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe458047dc0 con 0x7fe46806cdc0 2024-06-09T22:00:32.694 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.710+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0de55946-8987-4376-b9f9-f4957200232f", "id": 6} v 0) -- 0x7fe468154d00 con 0x7fe46806cdc0 2024-06-09T22:00:32.759 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.775+0000 7fe456ffd640 1 -- 172.21.15.148:0/2183805422 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0de55946-8987-4376-b9f9-f4957200232f", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7fe458047460 con 0x7fe46806cdc0 2024-06-09T22:00:32.759 INFO:teuthology.orchestra.run.smithi148.stdout:6 2024-06-09T22:00:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe46806cdc0 msgr2=0x7fe4681525e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 --2- 172.21.15.148:0/2183805422 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe46806cdc0 0x7fe4681525e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe458009860 tx=0x7fe4580046c0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.760 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 shutdown_connections 2024-06-09T22:00:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 --2- 172.21.15.148:0/2183805422 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe468151460 0x7fe468153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 --2- 172.21.15.148:0/2183805422 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4680c4c10 0x7fe46814d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 --2- 172.21.15.148:0/2183805422 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe46806cdc0 0x7fe4681525e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 >> 172.21.15.148:0/2183805422 conn(0x7fe468072ad0 msgr2=0x7fe4680c6470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 shutdown_connections 2024-06-09T22:00:32.761 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.776+0000 7fe46cd6a640 1 -- 172.21.15.148:0/2183805422 wait complete. 2024-06-09T22:00:32.769 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:32.784+0000 7fa707bb8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:00:32.771 DEBUG:teuthology.orchestra.run.smithi148:> sudo ceph --cluster ceph osd new 0524e1f0-9c95-4d96-8d3f-e120b647bc83 7 2024-06-09T22:00:32.868 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:32.884+0000 7f4ccb930040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-09T22:00:32.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.887+0000 7f66e0872640 1 Processor -- start 2024-06-09T22:00:32.871 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.887+0000 7f66e0872640 1 -- start start 2024-06-09T22:00:32.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.888+0000 7f66e0872640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f66dc0cadf0 0x7f66dc06fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.888+0000 7f66e0872640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66dc15dd60 con 0x7f66dc0caa10 2024-06-09T22:00:32.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.888+0000 7f66e0872640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66dc15e040 con 0x7f66dc0c8760 2024-06-09T22:00:32.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.888+0000 7f66e0872640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f66dc15e320 con 0x7f66dc0cadf0 2024-06-09T22:00:32.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.888+0000 7f66dad9d640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f66dc0c8760 0x7f66dc1541b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.148:44594/0 (socket says 172.21.15.148:44594) 2024-06-09T22:00:32.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.888+0000 7f66dad9d640 1 -- 172.21.15.148:0/3227634909 learned_addr learned my addr 172.21.15.148:0/3227634909 (peer_addr_for_me v1:172.21.15.148:0/0) 2024-06-09T22:00:32.872 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.889+0000 7f66d3fff640 1 --2- 172.21.15.148:0/3227634909 >> v2:172.21.15.32:3300/0 conn(0x7f66dc0cadf0 0x7f66dc06fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.891+0000 7f66d9d9b640 1 -- 172.21.15.148:0/3227634909 <== mon.1 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2464492696 0 0) 0x7f66dc15e040 con 0x7f66dc0c8760 2024-06-09T22:00:32.875 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.891+0000 7f66d9d9b640 1 -- 172.21.15.148:0/3227634909 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f66c8003520 con 0x7f66dc0c8760 2024-06-09T22:00:32.877 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.894+0000 7f66d3fff640 1 -- 172.21.15.148:0/3227634909 >> v1:172.21.15.148:6789/0 conn(0x7f66dc0c8760 legacy=0x7f66dc1541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.894+0000 7f66d3fff640 1 -- 172.21.15.148:0/3227634909 >> v1:172.21.15.32:6789/0 conn(0x7f66dc0caa10 legacy=0x7f66dc070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.878 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.894+0000 7f66d3fff640 1 -- 172.21.15.148:0/3227634909 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66dc15e600 con 0x7f66dc0cadf0 2024-06-09T22:00:32.880 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.897+0000 7f66d3fff640 1 --2- 172.21.15.148:0/3227634909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0cadf0 0x7f66dc06fc20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f66c40097d0 tx=0x7f66c402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23d8725a1f76b4 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:32.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.900+0000 7f66d9d9b640 1 -- 172.21.15.148:0/3227634909 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c4037070 con 0x7f66dc0cadf0 2024-06-09T22:00:32.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.900+0000 7f66d9d9b640 1 -- 172.21.15.148:0/3227634909 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66c4032440 con 0x7f66dc0cadf0 2024-06-09T22:00:32.884 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.900+0000 7f66d9d9b640 1 -- 172.21.15.148:0/3227634909 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66c4032810 con 0x7f66dc0cadf0 2024-06-09T22:00:32.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.901+0000 7f66e0872640 1 -- 172.21.15.148:0/3227634909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0cadf0 msgr2=0x7f66dc06fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.901+0000 7f66e0872640 1 --2- 172.21.15.148:0/3227634909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0cadf0 0x7f66dc06fc20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f66c40097d0 tx=0x7f66c402a270 comp rx=0 tx=0).stop 2024-06-09T22:00:32.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.901+0000 7f66e0872640 1 -- 172.21.15.148:0/3227634909 shutdown_connections 2024-06-09T22:00:32.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.901+0000 7f66e0872640 1 --2- 172.21.15.148:0/3227634909 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0cadf0 0x7f66dc06fc20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.885 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.901+0000 7f66e0872640 1 -- 172.21.15.148:0/3227634909 >> 172.21.15.148:0/3227634909 conn(0x7f66dc062660 msgr2=0x7f66dc0717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:32.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.901+0000 7f66e0872640 1 -- 172.21.15.148:0/3227634909 shutdown_connections 2024-06-09T22:00:32.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.901+0000 7f66e0872640 1 -- 172.21.15.148:0/3227634909 wait complete. 2024-06-09T22:00:32.886 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.903+0000 7f66e0872640 1 Processor -- start 2024-06-09T22:00:32.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.903+0000 7f66e0872640 1 -- start start 2024-06-09T22:00:32.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.903+0000 7f66e0872640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0c8760 0x7f66dc146b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.903+0000 7f66e0872640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66dc0caa10 0x7f66dc1f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.903+0000 7f66e0872640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f66dc0cadf0 0x7f66dc1f3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:32.887 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.903+0000 7f66e0872640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc1548c0 con 0x7f66dc0c8760 2024-06-09T22:00:32.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.903+0000 7f66e0872640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f66dc071320 con 0x7f66dc0caa10 2024-06-09T22:00:32.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.903+0000 7f66e0872640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f66dc154a40 con 0x7f66dc0cadf0 2024-06-09T22:00:32.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.904+0000 7f66d3fff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66dc0caa10 0x7f66dc1f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.904+0000 7f66dad9d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f66dc0cadf0 0x7f66dc1f3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.904+0000 7f66da59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0c8760 0x7f66dc146b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:32.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.904+0000 7f66da59c640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0c8760 0x7f66dc146b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.148:46254/0 (socket says 172.21.15.148:46254) 2024-06-09T22:00:32.888 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.904+0000 7f66da59c640 1 -- 172.21.15.148:0/1786000103 learned_addr learned my addr 172.21.15.148:0/1786000103 (peer_addr_for_me v2:172.21.15.148:0/0) 2024-06-09T22:00:32.890 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:32.906+0000 7fa707bb8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:00:32.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.908+0000 7f66da59c640 1 -- 172.21.15.148:0/1786000103 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f66dc0cadf0 msgr2=0x7f66dc1f3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.908+0000 7f66da59c640 1 --2- 172.21.15.148:0/1786000103 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f66dc0cadf0 0x7f66dc1f3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.908+0000 7f66da59c640 1 -- 172.21.15.148:0/1786000103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66dc0caa10 msgr2=0x7f66dc1f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:32.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.908+0000 7f66da59c640 1 --2- 172.21.15.148:0/1786000103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66dc0caa10 0x7f66dc1f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:32.892 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.908+0000 7f66da59c640 1 -- 172.21.15.148:0/1786000103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66dc14d250 con 0x7f66dc0c8760 2024-06-09T22:00:32.894 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.910+0000 7f66da59c640 1 --2- 172.21.15.148:0/1786000103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0c8760 0x7f66dc146b50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00c430 tx=0x7f66cc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:32.897 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.913+0000 7f66d2ffd640 1 -- 172.21.15.148:0/1786000103 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc0052d0 con 0x7f66dc0c8760 2024-06-09T22:00:32.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.913+0000 7f66d2ffd640 1 -- 172.21.15.148:0/1786000103 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66cc005490 con 0x7f66dc0c8760 2024-06-09T22:00:32.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.913+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66dc155e40 con 0x7f66dc0c8760 2024-06-09T22:00:32.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.913+0000 7f66d2ffd640 1 -- 172.21.15.148:0/1786000103 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66cc007260 con 0x7f66dc0c8760 2024-06-09T22:00:32.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.913+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66dc156100 con 0x7f66dc0c8760 2024-06-09T22:00:32.898 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.915+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66dc074f90 con 0x7f66dc0c8760 2024-06-09T22:00:32.900 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.917+0000 7f66d2ffd640 1 -- 172.21.15.148:0/1786000103 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f66cc007400 con 0x7f66dc0c8760 2024-06-09T22:00:32.901 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:32.917+0000 7f66d2ffd640 1 -- 172.21.15.148:0/1786000103 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f66cc011bf0 con 0x7f66dc0c8760 2024-06-09T22:00:32.990 INFO:tasks.ceph.mgr.x.smithi148.stderr:2024-06-09T22:00:33.006+0000 7f4ccb930040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-09T22:00:33.038 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.054+0000 7f66d2ffd640 1 -- 172.21.15.148:0/1786000103 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f66cc0035f0 con 0x7f66dc0c8760 2024-06-09T22:00:33.108 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.124+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd new", "uuid": "0524e1f0-9c95-4d96-8d3f-e120b647bc83", "id": 7} v 0) -- 0x7f66dc15da70 con 0x7f66dc0c8760 2024-06-09T22:00:33.260 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.276+0000 7f66d2ffd640 1 -- 172.21.15.148:0/1786000103 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 2) ==== 45274+0+0 (secure 0 0 0) 0x7f66cc003a50 con 0x7f66dc0c8760 2024-06-09T22:00:33.426 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.442+0000 7f66d2ffd640 1 -- 172.21.15.148:0/1786000103 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0524e1f0-9c95-4d96-8d3f-e120b647bc83", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f66cc01f7a0 con 0x7f66dc0c8760 2024-06-09T22:00:33.427 INFO:teuthology.orchestra.run.smithi148.stdout:7 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.443+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0c8760 msgr2=0x7f66dc146b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.443+0000 7f66e0872640 1 --2- 172.21.15.148:0/1786000103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0c8760 0x7f66dc146b50 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f66cc00c430 tx=0x7f66cc00c8f0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.443+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 shutdown_connections 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.443+0000 7f66e0872640 1 --2- 172.21.15.148:0/1786000103 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f66dc0cadf0 0x7f66dc1f3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.443+0000 7f66e0872640 1 --2- 172.21.15.148:0/1786000103 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f66dc0caa10 0x7f66dc1f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.443+0000 7f66e0872640 1 --2- 172.21.15.148:0/1786000103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66dc0c8760 0x7f66dc146b50 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.443+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 >> 172.21.15.148:0/1786000103 conn(0x7f66dc062660 msgr2=0x7f66dc071710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.444+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 shutdown_connections 2024-06-09T22:00:33.428 INFO:teuthology.orchestra.run.smithi148.stderr:2024-06-09T22:00:33.444+0000 7f66e0872640 1 -- 172.21.15.148:0/1786000103 wait complete. 2024-06-09T22:00:33.439 INFO:tasks.ceph.osd.0:Restarting daemon 2024-06-09T22:00:33.439 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 0 2024-06-09T22:00:33.441 INFO:tasks.ceph.osd.0:Started 2024-06-09T22:00:33.441 INFO:tasks.ceph.osd.1:Restarting daemon 2024-06-09T22:00:33.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 1 2024-06-09T22:00:33.443 INFO:tasks.ceph.osd.1:Started 2024-06-09T22:00:33.443 INFO:tasks.ceph.osd.2:Restarting daemon 2024-06-09T22:00:33.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 2 2024-06-09T22:00:33.445 INFO:tasks.ceph.osd.2:Started 2024-06-09T22:00:33.446 INFO:tasks.ceph.osd.3:Restarting daemon 2024-06-09T22:00:33.446 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 3 2024-06-09T22:00:33.448 INFO:tasks.ceph.osd.3:Started 2024-06-09T22:00:33.448 INFO:tasks.ceph.osd.4:Restarting daemon 2024-06-09T22:00:33.448 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 4 2024-06-09T22:00:33.451 INFO:tasks.ceph.osd.4:Started 2024-06-09T22:00:33.452 INFO:tasks.ceph.osd.5:Restarting daemon 2024-06-09T22:00:33.452 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 5 2024-06-09T22:00:33.454 INFO:tasks.ceph.osd.5:Started 2024-06-09T22:00:33.454 INFO:tasks.ceph.osd.6:Restarting daemon 2024-06-09T22:00:33.454 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 6 2024-06-09T22:00:33.456 INFO:tasks.ceph.osd.6:Started 2024-06-09T22:00:33.456 INFO:tasks.ceph.osd.7:Restarting daemon 2024-06-09T22:00:33.456 DEBUG:teuthology.orchestra.run.smithi148:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-osd -f --cluster ceph -i 7 2024-06-09T22:00:33.458 INFO:tasks.ceph.osd.7:Started 2024-06-09T22:00:33.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-09T22:00:33.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.601+0000 7fa6c536a640 1 Processor -- start 2024-06-09T22:00:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.601+0000 7fa6c536a640 1 -- start start 2024-06-09T22:00:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.602+0000 7fa6c536a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa6c0152820 0x7fa6c00c9e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.602+0000 7fa6c536a640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fa6c00cdbe0 0x7fa6c00ce070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.602+0000 7fa6c536a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6c00c3380 con 0x7fa6c0158f70 2024-06-09T22:00:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.602+0000 7fa6c536a640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7fa6c00c90d0 con 0x7fa6c0152820 2024-06-09T22:00:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.602+0000 7fa6c536a640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c00c93b0 con 0x7fa6c00cdbe0 2024-06-09T22:00:33.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.612+0000 7fa6beffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa6c0152820 0x7fa6c00c9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:33.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.613+0000 7fa6beffd640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fa6c0152820 0x7fa6c00c9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:58610/0 (socket says 172.21.15.32:58610) 2024-06-09T22:00:33.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.613+0000 7fa6beffd640 1 -- 172.21.15.32:0/3805573028 learned_addr learned my addr 172.21.15.32:0/3805573028 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:33.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.621+0000 7fa6beffd640 1 -- 172.21.15.32:0/3805573028 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fa6c00cdbe0 msgr2=0x7fa6c00ce070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:33.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.621+0000 7fa6beffd640 1 --2- 172.21.15.32:0/3805573028 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fa6c00cdbe0 0x7fa6c00ce070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.621+0000 7fa6beffd640 1 -- 172.21.15.32:0/3805573028 >> v1:172.21.15.148:6789/0 conn(0x7fa6c0158f70 legacy=0x7fa6c00c2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:33.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.621+0000 7fa6beffd640 1 -- 172.21.15.32:0/3805573028 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c015b960 con 0x7fa6c0152820 2024-06-09T22:00:33.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.624+0000 7fa6beffd640 1 --2- 172.21.15.32:0/3805573028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c00c9e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009870 tx=0x7fa6b002a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec1153a11ad153d2 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:33.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.629+0000 7fa6bdffb640 1 -- 172.21.15.32:0/3805573028 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0037070 con 0x7fa6c0152820 2024-06-09T22:00:33.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.629+0000 7fa6bdffb640 1 -- 172.21.15.32:0/3805573028 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6b002ae50 con 0x7fa6c0152820 2024-06-09T22:00:33.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6bdffb640 1 -- 172.21.15.32:0/3805573028 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b0032860 con 0x7fa6c0152820 2024-06-09T22:00:33.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6c536a640 1 -- 172.21.15.32:0/3805573028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 msgr2=0x7fa6c00c9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:33.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6c536a640 1 --2- 172.21.15.32:0/3805573028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c00c9e50 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa6b0009870 tx=0x7fa6b002a300 comp rx=0 tx=0).stop 2024-06-09T22:00:33.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6c536a640 1 -- 172.21.15.32:0/3805573028 shutdown_connections 2024-06-09T22:00:33.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6c536a640 1 --2- 172.21.15.32:0/3805573028 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fa6c00cdbe0 0x7fa6c00ce070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6c536a640 1 --2- 172.21.15.32:0/3805573028 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c00c9e50 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6c536a640 1 -- 172.21.15.32:0/3805573028 >> 172.21.15.32:0/3805573028 conn(0x7fa6c0061de0 msgr2=0x7fa6c00621c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:33.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6c536a640 1 -- 172.21.15.32:0/3805573028 shutdown_connections 2024-06-09T22:00:33.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.630+0000 7fa6c536a640 1 -- 172.21.15.32:0/3805573028 wait complete. 2024-06-09T22:00:33.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.632+0000 7fa6c536a640 1 Processor -- start 2024-06-09T22:00:33.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.632+0000 7fa6c536a640 1 -- start start 2024-06-09T22:00:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.632+0000 7fa6c536a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c00cdbe0 0x7fa6c01e3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.632+0000 7fa6c536a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c01e2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.632+0000 7fa6c536a640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fa6c0158f70 0x7fa6c015af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.632+0000 7fa6c536a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c015c200 con 0x7fa6c00cdbe0 2024-06-09T22:00:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.632+0000 7fa6c536a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fa6c015bf20 con 0x7fa6c0152820 2024-06-09T22:00:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.632+0000 7fa6c536a640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fa6c00ca390 con 0x7fa6c0158f70 2024-06-09T22:00:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.633+0000 7fa6be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c01e2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:33.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.633+0000 7fa6beffd640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c00cdbe0 0x7fa6c01e3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:33.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.633+0000 7fa6be7fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c01e2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:58618/0 (socket says 172.21.15.32:58618) 2024-06-09T22:00:33.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.633+0000 7fa6be7fc640 1 -- 172.21.15.32:0/1914276254 learned_addr learned my addr 172.21.15.32:0/1914276254 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:33.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.636+0000 7fa6be7fc640 1 -- 172.21.15.32:0/1914276254 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fa6c0158f70 msgr2=0x7fa6c015af00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:33.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.636+0000 7fa6be7fc640 1 --2- 172.21.15.32:0/1914276254 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fa6c0158f70 0x7fa6c015af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.636+0000 7fa6be7fc640 1 -- 172.21.15.32:0/1914276254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c00cdbe0 msgr2=0x7fa6c01e3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:33.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.636+0000 7fa6be7fc640 1 --2- 172.21.15.32:0/1914276254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c00cdbe0 0x7fa6c01e3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.636+0000 7fa6be7fc640 1 -- 172.21.15.32:0/1914276254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6c01e3ee0 con 0x7fa6c0152820 2024-06-09T22:00:33.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.639+0000 7fa6be7fc640 1 --2- 172.21.15.32:0/1914276254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c01e2960 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c4b0 tx=0x7fa6b400c970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.642+0000 7fa6a77fe640 1 -- 172.21.15.32:0/1914276254 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b40052d0 con 0x7fa6c0152820 2024-06-09T22:00:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.642+0000 7fa6a77fe640 1 -- 172.21.15.32:0/1914276254 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6b4005470 con 0x7fa6c0152820 2024-06-09T22:00:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.642+0000 7fa6a77fe640 1 -- 172.21.15.32:0/1914276254 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4012870 con 0x7fa6c0152820 2024-06-09T22:00:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.642+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6c01e3c00 con 0x7fa6c0152820 2024-06-09T22:00:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.643+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6c01ee990 con 0x7fa6c0152820 2024-06-09T22:00:33.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.643+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6c0074f90 con 0x7fa6c0152820 2024-06-09T22:00:33.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.757+0000 7fa6a77fe640 1 -- 172.21.15.32:0/1914276254 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 2) ==== 45274+0+0 (secure 0 0 0) 0x7fa6b4012a10 con 0x7fa6c0152820 2024-06-09T22:00:33.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.764+0000 7fa6a77fe640 1 -- 172.21.15.32:0/1914276254 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fa6b4003d00 con 0x7fa6c0152820 2024-06-09T22:00:33.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.764+0000 7fa6a77fe640 1 -- 172.21.15.32:0/1914276254 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa6b40037e0 con 0x7fa6c0152820 2024-06-09T22:00:33.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.846+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6c014c8a0 con 0x7fa6c0152820 2024-06-09T22:00:33.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.901+0000 7fa6a77fe640 1 -- 172.21.15.32:0/1914276254 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fa6b401a360 con 0x7fa6c0152820 2024-06-09T22:00:33.886 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:00:33.886 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":10,"fsid":"08d1d92e-3533-4051-bd41-97c73d8b6b19","created":"2024-06-09T22:00:27.411675+0000","modified":"2024-06-09T22:00:33.132294+0000","last_up_change":"0.000000","last_in_change":"2024-06-09T22:00:33.132294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4cc9d666-a633-4740-a293-be07ab208937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8db4805f-af3c-41a5-8eaf-1485c4ec8a5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"172c9719-1b63-460d-ba3e-dba05f194990","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"611e5921-1524-4cc9-9f33-a51d431115f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b85f7437-19ea-4f08-8a71-ef14330e0351","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae110414-eb0e-4253-91f2-0b2e83dad115","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0de55946-8987-4376-b9f9-f4957200232f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0524e1f0-9c95-4d96-8d3f-e120b647bc83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:00:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 msgr2=0x7fa6c01e2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 --2- 172.21.15.32:0/1914276254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c01e2960 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c4b0 tx=0x7fa6b400c970 comp rx=0 tx=0).stop 2024-06-09T22:00:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 shutdown_connections 2024-06-09T22:00:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 --2- 172.21.15.32:0/1914276254 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fa6c0158f70 0x7fa6c015af00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 --2- 172.21.15.32:0/1914276254 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fa6c0152820 0x7fa6c01e2960 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 --2- 172.21.15.32:0/1914276254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6c00cdbe0 0x7fa6c01e3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:33.887 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 >> 172.21.15.32:0/1914276254 conn(0x7fa6c0061de0 msgr2=0x7fa6c00cd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 shutdown_connections 2024-06-09T22:00:33.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:33.902+0000 7fa6c536a640 1 -- 172.21.15.32:0/1914276254 wait complete. 2024-06-09T22:00:33.899 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-06-09T22:00:33.899 INFO:tasks.ceph:Waiting for OSDs to come up 2024-06-09T22:00:34.062 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-06-09T22:00:34.078+0000 7f8c63f55640 -1 Falling back to public interface 2024-06-09T22:00:34.085 INFO:tasks.ceph.osd.4.smithi148.stderr:2024-06-09T22:00:34.100+0000 7fd8ce5dc640 -1 Falling back to public interface 2024-06-09T22:00:34.086 INFO:tasks.ceph.osd.5.smithi148.stderr:2024-06-09T22:00:34.101+0000 7f8f94019640 -1 Falling back to public interface 2024-06-09T22:00:34.087 INFO:tasks.ceph.osd.6.smithi148.stderr:2024-06-09T22:00:34.101+0000 7f9a1ac18640 -1 Falling back to public interface 2024-06-09T22:00:34.087 INFO:tasks.ceph.osd.6.smithi148.stderr:2024-06-09T22:00:34.102+0000 7f9a19599640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2024-06-09T22:00:34.088 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-06-09T22:00:34.105+0000 7f2280633640 -1 Falling back to public interface 2024-06-09T22:00:34.105 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-06-09T22:00:34.121+0000 7fe0caeb1640 -1 Falling back to public interface 2024-06-09T22:00:34.117 INFO:tasks.ceph.osd.7.smithi148.stderr:2024-06-09T22:00:34.132+0000 7fbc66b53640 -1 Falling back to public interface 2024-06-09T22:00:34.131 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-06-09T22:00:34.147+0000 7f424a9ba640 -1 Falling back to public interface 2024-06-09T22:00:34.658 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:34.674+0000 7fa6a8faf640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-09T22:00:34.705 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-09T22:00:34.720 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:00:34.737+0000 7fa6a8faf640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-06-09T22:00:34.799 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.815+0000 7f6717dc8640 1 Processor -- start 2024-06-09T22:00:34.799 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.816+0000 7f6717dc8640 1 -- start start 2024-06-09T22:00:34.800 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.816+0000 7f6717dc8640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f67101549c0 0x7f67100ccab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:34.800 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.816+0000 7f6717dc8640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f671015e6b0 0x7f671015af40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:34.800 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.816+0000 7f6717dc8640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f671015b480 0x7f67100ce510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:34.800 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.816+0000 7f6717dc8640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f67100cf1c0 con 0x7f671015b480 2024-06-09T22:00:34.801 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.816+0000 7f6717dc8640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7f671015a760 con 0x7f67101549c0 2024-06-09T22:00:34.801 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.816+0000 7f6717dc8640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f67100ca810 con 0x7f671015e6b0 2024-06-09T22:00:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.827+0000 7f6715b64640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f67101549c0 0x7f67100ccab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.828+0000 7f6715b64640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f67101549c0 0x7f67100ccab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:58680/0 (socket says 172.21.15.32:58680) 2024-06-09T22:00:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.828+0000 7f6715b64640 1 -- 172.21.15.32:0/1499409413 learned_addr learned my addr 172.21.15.32:0/1499409413 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:34.814 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.830+0000 7f6715b64640 1 -- 172.21.15.32:0/1499409413 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f671015e6b0 msgr2=0x7f671015af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:34.814 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.830+0000 7f6715b64640 1 --2- 172.21.15.32:0/1499409413 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f671015e6b0 0x7f671015af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:34.814 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.830+0000 7f6715b64640 1 -- 172.21.15.32:0/1499409413 >> v2:172.21.15.32:3300/0 conn(0x7f671015b480 msgr2=0x7f67100ce510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:34.814 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.830+0000 7f6715b64640 1 --2- 172.21.15.32:0/1499409413 >> v2:172.21.15.32:3300/0 conn(0x7f671015b480 0x7f67100ce510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:34.814 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.830+0000 7f6715b64640 1 -- 172.21.15.32:0/1499409413 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67100caaf0 con 0x7f67101549c0 2024-06-09T22:00:34.816 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.832+0000 7f6715b64640 1 --2- 172.21.15.32:0/1499409413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f67100ccab0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f67100d0e80 tx=0x7f670c02a290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a2d0991bcebedb6 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:34.817 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.834+0000 7f6714b62640 1 -- 172.21.15.32:0/1499409413 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c037070 con 0x7f67101549c0 2024-06-09T22:00:34.819 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.835+0000 7f6714b62640 1 -- 172.21.15.32:0/1499409413 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670c032440 con 0x7f67101549c0 2024-06-09T22:00:34.820 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.835+0000 7f6717dc8640 1 -- 172.21.15.32:0/1499409413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 msgr2=0x7f67100ccab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:34.820 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.835+0000 7f6717dc8640 1 --2- 172.21.15.32:0/1499409413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f67100ccab0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f67100d0e80 tx=0x7f670c02a290 comp rx=0 tx=0).stop 2024-06-09T22:00:34.820 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.836+0000 7f6717dc8640 1 -- 172.21.15.32:0/1499409413 shutdown_connections 2024-06-09T22:00:34.820 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.836+0000 7f6717dc8640 1 --2- 172.21.15.32:0/1499409413 >> v2:172.21.15.32:3300/0 conn(0x7f671015b480 0x7f67100ce510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:34.820 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.836+0000 7f6717dc8640 1 --2- 172.21.15.32:0/1499409413 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f671015e6b0 0x7f671015af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:34.820 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.836+0000 7f6717dc8640 1 --2- 172.21.15.32:0/1499409413 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f67100ccab0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:34.820 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.836+0000 7f6717dc8640 1 -- 172.21.15.32:0/1499409413 >> 172.21.15.32:0/1499409413 conn(0x7f6710072ad0 msgr2=0x7f6710072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:34.820 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.836+0000 7f6717dc8640 1 -- 172.21.15.32:0/1499409413 shutdown_connections 2024-06-09T22:00:34.821 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.836+0000 7f6717dc8640 1 -- 172.21.15.32:0/1499409413 wait complete. 2024-06-09T22:00:34.821 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.837+0000 7f6717dc8640 1 Processor -- start 2024-06-09T22:00:34.822 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.837+0000 7f6717dc8640 1 -- start start 2024-06-09T22:00:34.822 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.837+0000 7f6717dc8640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f6710151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:34.822 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.838+0000 7f6717dc8640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f671015b480 0x7f67100ce6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:34.822 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.838+0000 7f6717dc8640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e6b0 0x7f67101f3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:34.822 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.838+0000 7f6717dc8640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67100cd1d0 con 0x7f671015e6b0 2024-06-09T22:00:34.822 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.838+0000 7f6717dc8640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f671015aa20 con 0x7f67101549c0 2024-06-09T22:00:34.822 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.838+0000 7f6717dc8640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f671015ead0 con 0x7f671015b480 2024-06-09T22:00:34.823 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.838+0000 7f6715b64640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f6710151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:34.823 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.838+0000 7f6715b64640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f6710151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:58686/0 (socket says 172.21.15.32:58686) 2024-06-09T22:00:34.823 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.838+0000 7f6715b64640 1 -- 172.21.15.32:0/4120821735 learned_addr learned my addr 172.21.15.32:0/4120821735 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:34.826 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.842+0000 7f6715b64640 1 -- 172.21.15.32:0/4120821735 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f671015b480 msgr2=0x7f67100ce6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:34.826 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.842+0000 7f6715b64640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f671015b480 0x7f67100ce6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:34.826 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.842+0000 7f6715b64640 1 -- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e6b0 msgr2=0x7f67101f3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:34.826 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.842+0000 7f6715b64640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e6b0 0x7f67101f3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:34.826 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.842+0000 7f6715b64640 1 -- 172.21.15.32:0/4120821735 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67101524a0 con 0x7f67101549c0 2024-06-09T22:00:34.829 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.845+0000 7f6715b64640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f6710151d40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f670c000c00 tx=0x7f670c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:34.835 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.851+0000 7f67067fc640 1 -- 172.21.15.32:0/4120821735 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c041020 con 0x7f67101549c0 2024-06-09T22:00:34.835 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.851+0000 7f67067fc640 1 -- 172.21.15.32:0/4120821735 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f670c0325f0 con 0x7f67101549c0 2024-06-09T22:00:34.835 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.851+0000 7f67067fc640 1 -- 172.21.15.32:0/4120821735 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f670c037040 con 0x7f67101549c0 2024-06-09T22:00:34.835 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.851+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6710153090 con 0x7f67101549c0 2024-06-09T22:00:34.835 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.851+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6710153650 con 0x7f67101549c0 2024-06-09T22:00:34.836 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:34.852+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f671004f760 con 0x7f67101549c0 2024-06-09T22:00:35.115 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.131+0000 7f67067fc640 1 -- 172.21.15.32:0/4120821735 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 3) ==== 90336+0+0 (secure 0 0 0) 0x7f670c044020 con 0x7f67101549c0 2024-06-09T22:00:35.115 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.131+0000 7f67067fc640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f66f406bcd0 0x7f66f406e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:35.116 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.132+0000 7f6715363640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f66f406bcd0 0x7f66f406e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:35.116 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.132+0000 7f6715363640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f66f406bcd0 0x7f66f406e190 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f67100b5c70 tx=0x7f6700006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:35.127 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.143+0000 7f67067fc640 1 -- 172.21.15.32:0/4120821735 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f670c078ca0 con 0x7f67101549c0 2024-06-09T22:00:35.128 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.145+0000 7f67067fc640 1 -- 172.21.15.32:0/4120821735 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f670c04b030 con 0x7f67101549c0 2024-06-09T22:00:35.333 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.349+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6710154050 con 0x7f67101549c0 2024-06-09T22:00:35.350 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.363+0000 7f67067fc640 1 -- 172.21.15.32:0/4120821735 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f670c07a710 con 0x7f67101549c0 2024-06-09T22:00:35.350 INFO:teuthology.misc.health.smithi032.stdout: 2024-06-09T22:00:35.350 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"08d1d92e-3533-4051-bd41-97c73d8b6b19","created":"2024-06-09T22:00:27.411675+0000","modified":"2024-06-09T22:00:33.132294+0000","last_up_change":"0.000000","last_in_change":"2024-06-09T22:00:33.132294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"4cc9d666-a633-4740-a293-be07ab208937","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8db4805f-af3c-41a5-8eaf-1485c4ec8a5e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"172c9719-1b63-460d-ba3e-dba05f194990","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"611e5921-1524-4cc9-9f33-a51d431115f2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b85f7437-19ea-4f08-8a71-ef14330e0351","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae110414-eb0e-4253-91f2-0b2e83dad115","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"0de55946-8987-4376-b9f9-f4957200232f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0524e1f0-9c95-4d96-8d3f-e120b647bc83","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:00:35.351 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f66f406bcd0 msgr2=0x7f66f406e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:35.351 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f66f406bcd0 0x7f66f406e190 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f67100b5c70 tx=0x7f6700006c90 comp rx=0 tx=0).stop 2024-06-09T22:00:35.351 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 msgr2=0x7f6710151d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:35.351 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f6710151d40 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f670c000c00 tx=0x7f670c0046c0 comp rx=0 tx=0).stop 2024-06-09T22:00:35.351 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 shutdown_connections 2024-06-09T22:00:35.351 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f671015e6b0 0x7f67101f3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:35.352 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f66f406bcd0 0x7f66f406e190 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:35.352 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f671015b480 0x7f67100ce6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:35.352 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 --2- 172.21.15.32:0/4120821735 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f67101549c0 0x7f6710151d40 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:35.352 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 >> 172.21.15.32:0/4120821735 conn(0x7f6710072ad0 msgr2=0x7f6710150b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:35.352 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 shutdown_connections 2024-06-09T22:00:35.352 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:35.365+0000 7f6717dc8640 1 -- 172.21.15.32:0/4120821735 wait complete. 2024-06-09T22:00:35.360 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-09T22:00:37.510 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-06-09T22:00:37.526+0000 7f8c63f55640 -1 osd.1 0 log_to_monitors true 2024-06-09T22:00:37.541 INFO:tasks.ceph.osd.4.smithi148.stderr:2024-06-09T22:00:37.556+0000 7fd8ce5dc640 -1 osd.4 0 log_to_monitors true 2024-06-09T22:00:37.564 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-06-09T22:00:37.580+0000 7f2280633640 -1 osd.0 0 log_to_monitors true 2024-06-09T22:00:37.578 INFO:tasks.ceph.osd.6.smithi148.stderr:2024-06-09T22:00:37.593+0000 7f9a1ac18640 -1 osd.6 0 log_to_monitors true 2024-06-09T22:00:37.601 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-06-09T22:00:37.615+0000 7fe0caeb1640 -1 osd.2 0 log_to_monitors true 2024-06-09T22:00:37.605 INFO:tasks.ceph.osd.5.smithi148.stderr:2024-06-09T22:00:37.620+0000 7f8f94019640 -1 osd.5 0 log_to_monitors true 2024-06-09T22:00:37.630 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-06-09T22:00:37.646+0000 7f424a9ba640 -1 osd.3 0 log_to_monitors true 2024-06-09T22:00:37.631 INFO:tasks.ceph.osd.7.smithi148.stderr:2024-06-09T22:00:37.646+0000 7fbc66b53640 -1 osd.7 0 log_to_monitors true 2024-06-09T22:00:40.416 INFO:tasks.ceph.osd.5.smithi148.stderr:2024-06-09T22:00:40.431+0000 7f8f8f92c640 -1 osd.5 0 waiting for initial osdmap 2024-06-09T22:00:40.418 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-06-09T22:00:40.434+0000 7f227c394640 -1 osd.0 0 waiting for initial osdmap 2024-06-09T22:00:40.420 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-06-09T22:00:40.436+0000 7f8c606ff640 -1 osd.1 0 waiting for initial osdmap 2024-06-09T22:00:40.461 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-06-09T22:00:40.477+0000 7f8c57d0b640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:00:40.465 INFO:tasks.ceph.osd.7.smithi148.stderr:2024-06-09T22:00:40.480+0000 7fbc62ae9640 -1 osd.7 0 waiting for initial osdmap 2024-06-09T22:00:40.467 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-06-09T22:00:40.482+0000 7fe0c6e47640 -1 osd.2 0 waiting for initial osdmap 2024-06-09T22:00:40.479 INFO:tasks.ceph.osd.5.smithi148.stderr:2024-06-09T22:00:40.494+0000 7f8f8774c640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:00:40.493 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-06-09T22:00:40.509+0000 7f22739b1640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:00:40.497 INFO:tasks.ceph.osd.7.smithi148.stderr:2024-06-09T22:00:40.512+0000 7fbc5a909640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:00:40.502 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-06-09T22:00:40.518+0000 7f4247164640 -1 osd.3 0 waiting for initial osdmap 2024-06-09T22:00:40.508 INFO:tasks.ceph.osd.6.smithi148.stderr:2024-06-09T22:00:40.524+0000 7f9a16580640 -1 osd.6 0 waiting for initial osdmap 2024-06-09T22:00:40.539 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-06-09T22:00:40.555+0000 7f423e770640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:00:40.548 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-06-09T22:00:40.565+0000 7fe0bec67640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:00:40.550 INFO:tasks.ceph.osd.6.smithi148.stderr:2024-06-09T22:00:40.566+0000 7f9a0e3a0640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:00:40.567 INFO:tasks.ceph.osd.4.smithi148.stderr:2024-06-09T22:00:40.582+0000 7fd8ca572640 -1 osd.4 0 waiting for initial osdmap 2024-06-09T22:00:40.701 INFO:tasks.ceph.osd.4.smithi148.stderr:2024-06-09T22:00:40.716+0000 7fd8c2392640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-09T22:00:41.707 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:00:41.710 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:00:42.166 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-09T22:00:42.264 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.280+0000 7efc19990640 1 Processor -- start 2024-06-09T22:00:42.265 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.281+0000 7efc19990640 1 -- start start 2024-06-09T22:00:42.265 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.281+0000 7efc19990640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc140c9ca0 0x7efc14165ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:42.265 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.281+0000 7efc19990640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc14072b30 con 0x7efc140c88c0 2024-06-09T22:00:42.265 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.281+0000 7efc19990640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc14072e10 con 0x7efc14157e10 2024-06-09T22:00:42.266 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.281+0000 7efc19990640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7efc140c7de0 con 0x7efc140c9ca0 2024-06-09T22:00:42.266 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.282+0000 7efc137fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc140c9ca0 0x7efc14165ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:42.266 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.282+0000 7efc12ffd640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7efc14157e10 0x7efc1415a600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.32:41600/0 (socket says 172.21.15.32:41600) 2024-06-09T22:00:42.266 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.282+0000 7efc12ffd640 1 -- 172.21.15.32:0/4006292526 learned_addr learned my addr 172.21.15.32:0/4006292526 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:00:42.269 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.285+0000 7efc137fe640 1 -- 172.21.15.32:0/4006292526 >> v1:172.21.15.148:6789/0 conn(0x7efc14157e10 legacy=0x7efc1415a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:42.269 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.285+0000 7efc137fe640 1 -- 172.21.15.32:0/4006292526 >> v1:172.21.15.32:6789/0 conn(0x7efc140c88c0 legacy=0x7efc140c9590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:42.269 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.285+0000 7efc137fe640 1 -- 172.21.15.32:0/4006292526 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc140c80c0 con 0x7efc140c9ca0 2024-06-09T22:00:42.271 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.287+0000 7efc137fe640 1 --2- 172.21.15.32:0/4006292526 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc140c9ca0 0x7efc14165ed0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009dc0 tx=0x7efc0c02a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=821393f59e61b66f server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:42.274 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc127fc640 1 -- 172.21.15.32:0/4006292526 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c02ae60 con 0x7efc140c9ca0 2024-06-09T22:00:42.275 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc127fc640 1 -- 172.21.15.32:0/4006292526 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc0c007740 con 0x7efc140c9ca0 2024-06-09T22:00:42.275 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc127fc640 1 -- 172.21.15.32:0/4006292526 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c007b10 con 0x7efc140c9ca0 2024-06-09T22:00:42.275 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc19990640 1 -- 172.21.15.32:0/4006292526 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc140c9ca0 msgr2=0x7efc14165ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:42.276 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc19990640 1 --2- 172.21.15.32:0/4006292526 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc140c9ca0 0x7efc14165ed0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7efc0c009dc0 tx=0x7efc0c02a680 comp rx=0 tx=0).stop 2024-06-09T22:00:42.276 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc19990640 1 -- 172.21.15.32:0/4006292526 shutdown_connections 2024-06-09T22:00:42.276 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc19990640 1 --2- 172.21.15.32:0/4006292526 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc140c9ca0 0x7efc14165ed0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:42.276 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc19990640 1 -- 172.21.15.32:0/4006292526 >> 172.21.15.32:0/4006292526 conn(0x7efc1406cae0 msgr2=0x7efc14070630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:42.276 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc19990640 1 -- 172.21.15.32:0/4006292526 shutdown_connections 2024-06-09T22:00:42.276 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.291+0000 7efc19990640 1 -- 172.21.15.32:0/4006292526 wait complete. 2024-06-09T22:00:42.276 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.293+0000 7efc19990640 1 Processor -- start 2024-06-09T22:00:42.276 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.293+0000 7efc19990640 1 -- start start 2024-06-09T22:00:42.277 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.293+0000 7efc19990640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc140c88c0 0x7efc141f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:42.278 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.293+0000 7efc19990640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc140c9ca0 0x7efc14156d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:42.278 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.293+0000 7efc19990640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc14157e10 0x7efc1414d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:42.278 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.293+0000 7efc19990640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efc14073830 con 0x7efc140c88c0 2024-06-09T22:00:42.278 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.293+0000 7efc19990640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efc14073320 con 0x7efc140c9ca0 2024-06-09T22:00:42.278 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.293+0000 7efc19990640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7efc140739b0 con 0x7efc14157e10 2024-06-09T22:00:42.279 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.294+0000 7efc0bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc140c9ca0 0x7efc14156d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:42.279 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.294+0000 7efc137fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc14157e10 0x7efc1414d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:42.279 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.295+0000 7efc137fe640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc14157e10 0x7efc1414d060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.32:55128/0 (socket says 172.21.15.32:55128) 2024-06-09T22:00:42.279 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.295+0000 7efc137fe640 1 -- 172.21.15.32:0/3921516419 learned_addr learned my addr 172.21.15.32:0/3921516419 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:42.279 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.295+0000 7efc0bfff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc140c9ca0 0x7efc14156d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:60156/0 (socket says 172.21.15.32:60156) 2024-06-09T22:00:42.283 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.299+0000 7efc137fe640 1 -- 172.21.15.32:0/3921516419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc140c9ca0 msgr2=0x7efc14156d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:42.283 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.299+0000 7efc137fe640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc140c9ca0 0x7efc14156d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:42.283 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.299+0000 7efc137fe640 1 -- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc140c88c0 msgr2=0x7efc141f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:42.283 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.299+0000 7efc137fe640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc140c88c0 0x7efc141f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.299+0000 7efc137fe640 1 -- 172.21.15.32:0/3921516419 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc141f00c0 con 0x7efc14157e10 2024-06-09T22:00:42.286 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.302+0000 7efc137fe640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc14157e10 0x7efc1414d060 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efc0c02ac70 tx=0x7efc0c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:42.290 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.306+0000 7efc10ff9640 1 -- 172.21.15.32:0/3921516419 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c040070 con 0x7efc14157e10 2024-06-09T22:00:42.290 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.306+0000 7efc10ff9640 1 -- 172.21.15.32:0/3921516419 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc0c02ae60 con 0x7efc14157e10 2024-06-09T22:00:42.290 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.306+0000 7efc10ff9640 1 -- 172.21.15.32:0/3921516419 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc0c045460 con 0x7efc14157e10 2024-06-09T22:00:42.290 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.306+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc140cc2b0 con 0x7efc14157e10 2024-06-09T22:00:42.290 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.306+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc140cc870 con 0x7efc14157e10 2024-06-09T22:00:42.290 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.307+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc1404f760 con 0x7efc14157e10 2024-06-09T22:00:42.308 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.325+0000 7efc10ff9640 1 -- 172.21.15.32:0/3921516419 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7efc0c039040 con 0x7efc14157e10 2024-06-09T22:00:42.309 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.325+0000 7efc10ff9640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efbf406bdf0 0x7efbf406e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:42.309 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.325+0000 7efc12ffd640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efbf406bdf0 0x7efbf406e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:42.309 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.325+0000 7efc12ffd640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efbf406bdf0 0x7efbf406e2b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efc04001550 tx=0x7efc04007a40 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:42.313 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.329+0000 7efc10ff9640 1 -- 172.21.15.32:0/3921516419 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(13..13 src has 1..13) ==== 2791+0+0 (secure 0 0 0) 0x7efc0c078e30 con 0x7efc14157e10 2024-06-09T22:00:42.745 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.761+0000 7efc10ff9640 1 -- 172.21.15.32:0/3921516419 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7efc141f00c0 con 0x7efc14157e10 2024-06-09T22:00:42.955 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:42.971+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc14073610 con 0x7efc14157e10 2024-06-09T22:00:43.068 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.084+0000 7efc10ff9640 1 -- 172.21.15.32:0/3921516419 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+12738 (secure 0 0 0) 0x7efc0c07b490 con 0x7efc14157e10 2024-06-09T22:00:43.069 INFO:teuthology.misc.health.smithi032.stdout: 2024-06-09T22:00:43.069 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":14,"fsid":"08d1d92e-3533-4051-bd41-97c73d8b6b19","created":"2024-06-09T22:00:27.411675+0000","modified":"2024-06-09T22:00:42.024015+0000","last_up_change":"2024-06-09T22:00:42.024015+0000","last_in_change":"2024-06-09T22:00:33.132294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:00:41.893545+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"4cc9d666-a633-4740-a293-be07ab208937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6811","nonce":3275688150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6813","nonce":3275688150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6817","nonce":3275688150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6815","nonce":3275688150}]},"public_addr":"172.21.15.32:6811/3275688150","cluster_addr":"172.21.15.32:6813/3275688150","heartbeat_back_addr":"172.21.15.32:6817/3275688150","heartbeat_front_addr":"172.21.15.32:6815/3275688150","state":["exists","up"]},{"osd":1,"uuid":"8db4805f-af3c-41a5-8eaf-1485c4ec8a5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6803","nonce":3057205177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6805","nonce":3057205177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6809","nonce":3057205177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6807","nonce":3057205177}]},"public_addr":"172.21.15.32:6803/3057205177","cluster_addr":"172.21.15.32:6805/3057205177","heartbeat_back_addr":"172.21.15.32:6809/3057205177","heartbeat_front_addr":"172.21.15.32:6807/3057205177","state":["exists","up"]},{"osd":2,"uuid":"172c9719-1b63-460d-ba3e-dba05f194990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6819","nonce":3248807238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6821","nonce":3248807238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6825","nonce":3248807238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6823","nonce":3248807238}]},"public_addr":"172.21.15.32:6819/3248807238","cluster_addr":"172.21.15.32:6821/3248807238","heartbeat_back_addr":"172.21.15.32:6825/3248807238","heartbeat_front_addr":"172.21.15.32:6823/3248807238","state":["exists","up"]},{"osd":3,"uuid":"611e5921-1524-4cc9-9f33-a51d431115f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6827","nonce":1835052153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6829","nonce":1835052153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6833","nonce":1835052153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6831","nonce":1835052153}]},"public_addr":"172.21.15.32:6827/1835052153","cluster_addr":"172.21.15.32:6829/1835052153","heartbeat_back_addr":"172.21.15.32:6833/1835052153","heartbeat_front_addr":"172.21.15.32:6831/1835052153","state":["exists","up"]},{"osd":4,"uuid":"b85f7437-19ea-4f08-8a71-ef14330e0351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6801","nonce":531504437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6803","nonce":531504437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6807","nonce":531504437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6805","nonce":531504437}]},"public_addr":"172.21.15.148:6801/531504437","cluster_addr":"172.21.15.148:6803/531504437","heartbeat_back_addr":"172.21.15.148:6807/531504437","heartbeat_front_addr":"172.21.15.148:6805/531504437","state":["exists","up"]},{"osd":5,"uuid":"ae110414-eb0e-4253-91f2-0b2e83dad115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6810","nonce":2398614348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6814","nonce":2398614348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6822","nonce":2398614348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6818","nonce":2398614348}]},"public_addr":"172.21.15.148:6810/2398614348","cluster_addr":"172.21.15.148:6814/2398614348","heartbeat_back_addr":"172.21.15.148:6822/2398614348","heartbeat_front_addr":"172.21.15.148:6818/2398614348","state":["exists","up"]},{"osd":6,"uuid":"0de55946-8987-4376-b9f9-f4957200232f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6811","nonce":975384648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6815","nonce":975384648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6821","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6823","nonce":975384648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6817","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6819","nonce":975384648}]},"public_addr":"172.21.15.148:6811/975384648","cluster_addr":"172.21.15.148:6815/975384648","heartbeat_back_addr":"172.21.15.148:6823/975384648","heartbeat_front_addr":"172.21.15.148:6819/975384648","state":["exists","up"]},{"osd":7,"uuid":"0524e1f0-9c95-4d96-8d3f-e120b647bc83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6825","nonce":2655085429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6827","nonce":2655085429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6831","nonce":2655085429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6829","nonce":2655085429}]},"public_addr":"172.21.15.148:6825/2655085429","cluster_addr":"172.21.15.148:6827/2655085429","heartbeat_back_addr":"172.21.15.148:6831/2655085429","heartbeat_front_addr":"172.21.15.148:6829/2655085429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.493643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.580349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:00:43.071 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efbf406bdf0 msgr2=0x7efbf406e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:43.071 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efbf406bdf0 0x7efbf406e2b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7efc04001550 tx=0x7efc04007a40 comp rx=0 tx=0).stop 2024-06-09T22:00:43.071 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc14157e10 msgr2=0x7efc1414d060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:43.071 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc14157e10 0x7efc1414d060 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7efc0c02ac70 tx=0x7efc0c004a40 comp rx=0 tx=0).stop 2024-06-09T22:00:43.071 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 shutdown_connections 2024-06-09T22:00:43.072 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efbf406bdf0 0x7efbf406e2b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.072 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc14157e10 0x7efc1414d060 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.072 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc140c9ca0 0x7efc14156d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.072 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 --2- 172.21.15.32:0/3921516419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc140c88c0 0x7efc141f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.072 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 >> 172.21.15.32:0/3921516419 conn(0x7efc1406cae0 msgr2=0x7efc140c2620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:43.072 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 shutdown_connections 2024-06-09T22:00:43.072 INFO:teuthology.misc.health.smithi032.stderr:2024-06-09T22:00:43.087+0000 7efc19990640 1 -- 172.21.15.32:0/3921516419 wait complete. 2024-06-09T22:00:43.082 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-06-09T22:00:43.082 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-06-09T22:00:43.082 DEBUG:tasks.ceph_manager:running mds.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-09T22:00:43.083 INFO:tasks.ceph.mds.a:Restarting daemon 2024-06-09T22:00:43.083 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a 2024-06-09T22:00:43.085 INFO:tasks.ceph.mds.a:Started 2024-06-09T22:00:43.085 DEBUG:tasks.ceph_manager:running mds.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-09T22:00:43.085 INFO:tasks.ceph.mds.c:Restarting daemon 2024-06-09T22:00:43.086 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c 2024-06-09T22:00:43.088 INFO:tasks.ceph.mds.c:Started 2024-06-09T22:00:43.088 DEBUG:tasks.ceph_manager:running mds.e under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.e.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-09T22:00:43.088 INFO:tasks.ceph.mds.e:Restarting daemon 2024-06-09T22:00:43.088 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i e 2024-06-09T22:00:43.090 INFO:tasks.ceph.mds.e:Started 2024-06-09T22:00:43.090 DEBUG:tasks.ceph_manager:running mds.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-09T22:00:43.090 INFO:tasks.ceph.mds.b:Restarting daemon 2024-06-09T22:00:43.090 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i b 2024-06-09T22:00:43.093 INFO:tasks.ceph.mds.b:Started 2024-06-09T22:00:43.093 DEBUG:tasks.ceph_manager:running mds.d under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.d.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-09T22:00:43.093 INFO:tasks.ceph.mds.d:Restarting daemon 2024-06-09T22:00:43.093 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.d.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i d 2024-06-09T22:00:43.095 INFO:tasks.ceph.mds.d:Started 2024-06-09T22:00:43.095 DEBUG:tasks.ceph_manager:running mds.f under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mds.f.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-09T22:00:43.096 INFO:tasks.ceph.mds.f:Restarting daemon 2024-06-09T22:00:43.096 DEBUG:teuthology.orchestra.run.smithi148:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.f.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i f 2024-06-09T22:00:43.098 INFO:tasks.ceph.mds.f:Started 2024-06-09T22:00:43.098 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-06-09T22:00:43.098 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-09T22:00:43.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.229+0000 7f45a483f640 1 Processor -- start 2024-06-09T22:00:43.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.229+0000 7f45a483f640 1 -- start start 2024-06-09T22:00:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.231+0000 7f45a483f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006d0e0 0x7f45a01542f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.231+0000 7f45a483f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a00c8650 con 0x7f45a00c9730 2024-06-09T22:00:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.231+0000 7f45a483f640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45a0070440 con 0x7f45a006f950 2024-06-09T22:00:43.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.231+0000 7f45a483f640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f45a006c620 con 0x7f45a006d0e0 2024-06-09T22:00:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.261+0000 7f459ed9d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006d0e0 0x7f45a01542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:43.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.262+0000 7f459ed9d640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006d0e0 0x7f45a01542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.32:55132/0 (socket says 172.21.15.32:55132) 2024-06-09T22:00:43.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.262+0000 7f459ed9d640 1 -- 172.21.15.32:0/4180511855 learned_addr learned my addr 172.21.15.32:0/4180511855 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.269+0000 7f459ed9d640 1 -- 172.21.15.32:0/4180511855 >> v1:172.21.15.148:6789/0 conn(0x7f45a006f950 legacy=0x7f45a0070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.269+0000 7f459ed9d640 1 -- 172.21.15.32:0/4180511855 >> v1:172.21.15.32:6789/0 conn(0x7f45a00c9730 legacy=0x7f45a006fd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:43.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.269+0000 7f459ed9d640 1 -- 172.21.15.32:0/4180511855 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a00c88f0 con 0x7f45a006d0e0 2024-06-09T22:00:43.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.271+0000 7f459ed9d640 1 --2- 172.21.15.32:0/4180511855 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006d0e0 0x7f45a01542f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4598009e80 tx=0x7f459802a780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2f485983b0f20df server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:43.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.276+0000 7f459d59a640 1 -- 172.21.15.32:0/4180511855 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598036070 con 0x7f45a006d0e0 2024-06-09T22:00:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.278+0000 7f459d59a640 1 -- 172.21.15.32:0/4180511855 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4598031600 con 0x7f45a006d0e0 2024-06-09T22:00:43.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.278+0000 7f459d59a640 1 -- 172.21.15.32:0/4180511855 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45980319d0 con 0x7f45a006d0e0 2024-06-09T22:00:43.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.279+0000 7f45a483f640 1 -- 172.21.15.32:0/4180511855 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006d0e0 msgr2=0x7f45a01542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:43.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.279+0000 7f45a483f640 1 --2- 172.21.15.32:0/4180511855 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006d0e0 0x7f45a01542f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f4598009e80 tx=0x7f459802a780 comp rx=0 tx=0).stop 2024-06-09T22:00:43.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.279+0000 7f45a483f640 1 -- 172.21.15.32:0/4180511855 shutdown_connections 2024-06-09T22:00:43.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.279+0000 7f45a483f640 1 --2- 172.21.15.32:0/4180511855 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006d0e0 0x7f45a01542f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.279+0000 7f45a483f640 1 -- 172.21.15.32:0/4180511855 >> 172.21.15.32:0/4180511855 conn(0x7f45a0071680 msgr2=0x7f45a0071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:43.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.279+0000 7f45a483f640 1 -- 172.21.15.32:0/4180511855 shutdown_connections 2024-06-09T22:00:43.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.279+0000 7f45a483f640 1 -- 172.21.15.32:0/4180511855 wait complete. 2024-06-09T22:00:43.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.281+0000 7f45a483f640 1 Processor -- start 2024-06-09T22:00:43.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.281+0000 7f45a483f640 1 -- start start 2024-06-09T22:00:43.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.282+0000 7f45a483f640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a006d0e0 0x7f45a01f0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.282+0000 7f45a483f640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006f950 0x7f45a015e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.282+0000 7f45a483f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45a00c9730 0x7f45a01e39c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.282+0000 7f45a483f640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45a006cbc0 con 0x7f45a00c9730 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.282+0000 7f45a483f640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f45a006c8e0 con 0x7f45a006d0e0 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.282+0000 7f45a483f640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f45a006d520 con 0x7f45a006f950 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.283+0000 7f459e59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a006d0e0 0x7f45a01f0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.283+0000 7f459dd9b640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006f950 0x7f45a015e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.283+0000 7f459e59c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a006d0e0 0x7f45a01f0f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:60162/0 (socket says 172.21.15.32:60162) 2024-06-09T22:00:43.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.283+0000 7f459e59c640 1 -- 172.21.15.32:0/3968965916 learned_addr learned my addr 172.21.15.32:0/3968965916 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:43.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.288+0000 7f459e59c640 1 -- 172.21.15.32:0/3968965916 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006f950 msgr2=0x7f45a015e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:43.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.288+0000 7f459e59c640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006f950 0x7f45a015e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.288+0000 7f459e59c640 1 -- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45a00c9730 msgr2=0x7f45a01e39c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:43.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.288+0000 7f459e59c640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45a00c9730 0x7f45a01e39c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.288+0000 7f459e59c640 1 -- 172.21.15.32:0/3968965916 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45a015b900 con 0x7f45a006d0e0 2024-06-09T22:00:43.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.288+0000 7f459dd9b640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006f950 0x7f45a015e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:00:43.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.292+0000 7f459e59c640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a006d0e0 0x7f45a01f0f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f459000c410 tx=0x7f459000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:43.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.296+0000 7f458effd640 1 -- 172.21.15.32:0/3968965916 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4590017070 con 0x7f45a006d0e0 2024-06-09T22:00:43.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.296+0000 7f458effd640 1 -- 172.21.15.32:0/3968965916 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45900035f0 con 0x7f45a006d0e0 2024-06-09T22:00:43.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.296+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45a01f14a0 con 0x7f45a006d0e0 2024-06-09T22:00:43.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.296+0000 7f458effd640 1 -- 172.21.15.32:0/3968965916 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45900039e0 con 0x7f45a006d0e0 2024-06-09T22:00:43.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.296+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45a0153270 con 0x7f45a006d0e0 2024-06-09T22:00:43.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.297+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45a00cd410 con 0x7f45a006d0e0 2024-06-09T22:00:43.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.321+0000 7f458effd640 1 -- 172.21.15.32:0/3968965916 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f459001a070 con 0x7f45a006d0e0 2024-06-09T22:00:43.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.322+0000 7f458effd640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f458006bdf0 0x7f458006e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:43.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.322+0000 7f459dd9b640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f458006bdf0 0x7f458006e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:43.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.322+0000 7f459dd9b640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f458006bdf0 0x7f458006e2b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f45a00cb340 tx=0x7f4594009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:43.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.325+0000 7f458effd640 1 -- 172.21.15.32:0/3968965916 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 4846+0+0 (secure 0 0 0) 0x7f459005f4e0 con 0x7f45a006d0e0 2024-06-09T22:00:43.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.635+0000 7f458effd640 1 -- 172.21.15.32:0/3968965916 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f45a00cd410 con 0x7f45a006d0e0 2024-06-09T22:00:43.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.921+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45a0068490 con 0x7f45a006d0e0 2024-06-09T22:00:43.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.941+0000 7f458effd640 1 -- 172.21.15.32:0/3968965916 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f4590021070 con 0x7f45a006d0e0 2024-06-09T22:00:43.926 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:00:43.926 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-06-09T22:00:43.926 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-06-09T22:00:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f458006bdf0 msgr2=0x7f458006e2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f458006bdf0 0x7f458006e2b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f45a00cb340 tx=0x7f4594009060 comp rx=0 tx=0).stop 2024-06-09T22:00:43.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a006d0e0 msgr2=0x7f45a01f0f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a006d0e0 0x7f45a01f0f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f459000c410 tx=0x7f459000c8d0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 shutdown_connections 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45a00c9730 0x7f45a01e39c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f458006bdf0 0x7f458006e2b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f45a006f950 0x7f45a015e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 --2- 172.21.15.32:0/3968965916 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f45a006d0e0 0x7f45a01f0f60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 >> 172.21.15.32:0/3968965916 conn(0x7f45a0071680 msgr2=0x7f45a00cc5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 shutdown_connections 2024-06-09T22:00:43.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:43.944+0000 7f45a483f640 1 -- 172.21.15.32:0/3968965916 wait complete. 2024-06-09T22:00:45.943 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-09T22:00:46.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.074+0000 7f0ff0483640 1 Processor -- start 2024-06-09T22:00:46.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.074+0000 7f0ff0483640 1 -- start start 2024-06-09T22:00:46.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.075+0000 7f0ff0483640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f0fe806f950 0x7f0fe806d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:46.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.075+0000 7f0ff0483640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0fe806fd30 0x7f0fe81542f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:46.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.075+0000 7f0ff0483640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fe80c8650 con 0x7f0fe80c9730 2024-06-09T22:00:46.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.075+0000 7f0ff0483640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe806d620 con 0x7f0fe806fd30 2024-06-09T22:00:46.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.075+0000 7f0ff0483640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7f0fe806c620 con 0x7f0fe806f950 2024-06-09T22:00:46.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.075+0000 7f0feea20640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0fe806fd30 0x7f0fe81542f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:46.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.076+0000 7f0feea20640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f0fe806fd30 0x7f0fe81542f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53952/0 (socket says 172.21.15.32:53952) 2024-06-09T22:00:46.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.076+0000 7f0feea20640 1 -- 172.21.15.32:0/2823982276 learned_addr learned my addr 172.21.15.32:0/2823982276 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:46.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.079+0000 7f0feea20640 1 -- 172.21.15.32:0/2823982276 >> v2:172.21.15.148:3300/0 conn(0x7f0fe806f950 msgr2=0x7f0fe806d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:46.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.079+0000 7f0feea20640 1 --2- 172.21.15.32:0/2823982276 >> v2:172.21.15.148:3300/0 conn(0x7f0fe806f950 0x7f0fe806d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:46.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.079+0000 7f0feea20640 1 -- 172.21.15.32:0/2823982276 >> v1:172.21.15.32:6789/0 conn(0x7f0fe80c9730 legacy=0x7f0fe8070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:46.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.079+0000 7f0feea20640 1 -- 172.21.15.32:0/2823982276 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe80c88f0 con 0x7f0fe806fd30 2024-06-09T22:00:46.067 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.083+0000 7f0feea20640 1 --2- 172.21.15.32:0/2823982276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806fd30 0x7f0fe81542f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8072230 tx=0x7f0fe002a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba973cd753dca961 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:46.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.089+0000 7f0fed21d640 1 -- 172.21.15.32:0/2823982276 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe0036070 con 0x7f0fe806fd30 2024-06-09T22:00:46.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.089+0000 7f0fed21d640 1 -- 172.21.15.32:0/2823982276 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fe0031600 con 0x7f0fe806fd30 2024-06-09T22:00:46.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.089+0000 7f0fed21d640 1 -- 172.21.15.32:0/2823982276 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fe00319f0 con 0x7f0fe806fd30 2024-06-09T22:00:46.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.090+0000 7f0ff0483640 1 -- 172.21.15.32:0/2823982276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806fd30 msgr2=0x7f0fe81542f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:46.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.090+0000 7f0ff0483640 1 --2- 172.21.15.32:0/2823982276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806fd30 0x7f0fe81542f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0fe8072230 tx=0x7f0fe002a7c0 comp rx=0 tx=0).stop 2024-06-09T22:00:46.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.091+0000 7f0ff0483640 1 -- 172.21.15.32:0/2823982276 shutdown_connections 2024-06-09T22:00:46.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.091+0000 7f0ff0483640 1 --2- 172.21.15.32:0/2823982276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806fd30 0x7f0fe81542f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:46.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.091+0000 7f0ff0483640 1 --2- 172.21.15.32:0/2823982276 >> v2:172.21.15.148:3300/0 conn(0x7f0fe806f950 0x7f0fe806d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:46.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.091+0000 7f0ff0483640 1 -- 172.21.15.32:0/2823982276 >> 172.21.15.32:0/2823982276 conn(0x7f0fe8071680 msgr2=0x7f0fe8071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:46.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.091+0000 7f0ff0483640 1 -- 172.21.15.32:0/2823982276 shutdown_connections 2024-06-09T22:00:46.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.091+0000 7f0ff0483640 1 -- 172.21.15.32:0/2823982276 wait complete. 2024-06-09T22:00:46.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.093+0000 7f0ff0483640 1 Processor -- start 2024-06-09T22:00:46.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.093+0000 7f0ff0483640 1 -- start start 2024-06-09T22:00:46.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.094+0000 7f0ff0483640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806f950 0x7f0fe814f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:46.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.094+0000 7f0ff0483640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0fe806fd30 0x7f0fe81f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:46.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.094+0000 7f0ff0483640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fe80c9730 0x7f0fe81f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:46.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.094+0000 7f0ff0483640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe806cbc0 con 0x7f0fe806f950 2024-06-09T22:00:46.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.094+0000 7f0ff0483640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f0fe806c8e0 con 0x7f0fe80c9730 2024-06-09T22:00:46.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.094+0000 7f0ff0483640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f0fe80701a0 con 0x7f0fe806fd30 2024-06-09T22:00:46.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.095+0000 7f0feda1e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0fe806fd30 0x7f0fe81f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:46.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.095+0000 7f0feea20640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fe80c9730 0x7f0fe81f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:46.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.095+0000 7f0fee21f640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806f950 0x7f0fe814f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:46.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.095+0000 7f0feda1e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0fe806fd30 0x7f0fe81f3940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.32:55166/0 (socket says 172.21.15.32:55166) 2024-06-09T22:00:46.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.095+0000 7f0feda1e640 1 -- 172.21.15.32:0/4190961955 learned_addr learned my addr 172.21.15.32:0/4190961955 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:46.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.100+0000 7f0feda1e640 1 -- 172.21.15.32:0/4190961955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fe80c9730 msgr2=0x7f0fe81f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:46.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.100+0000 7f0feda1e640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fe80c9730 0x7f0fe81f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:46.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.100+0000 7f0feda1e640 1 -- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806f950 msgr2=0x7f0fe814f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:46.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.100+0000 7f0feda1e640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806f950 0x7f0fe814f8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:46.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.100+0000 7f0feda1e640 1 -- 172.21.15.32:0/4190961955 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fe8157fa0 con 0x7f0fe806fd30 2024-06-09T22:00:46.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.103+0000 7f0feda1e640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0fe806fd30 0x7f0fe81f3940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800c400 tx=0x7f0fd800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:46.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.106+0000 7f0fdeffd640 1 -- 172.21.15.32:0/4190961955 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8017070 con 0x7f0fe806fd30 2024-06-09T22:00:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.106+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fe814f120 con 0x7f0fe806fd30 2024-06-09T22:00:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.106+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fe80b2cd0 con 0x7f0fe806fd30 2024-06-09T22:00:46.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.107+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fe80ceac0 con 0x7f0fe806fd30 2024-06-09T22:00:46.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.317+0000 7f0fdeffd640 1 -- 172.21.15.32:0/4190961955 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fd80052d0 con 0x7f0fe806fd30 2024-06-09T22:00:46.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.495+0000 7f0fdeffd640 1 -- 172.21.15.32:0/4190961955 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fd8021060 con 0x7f0fe806fd30 2024-06-09T22:00:46.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.497+0000 7f0fdeffd640 1 -- 172.21.15.32:0/4190961955 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 4) ==== 90442+0+0 (secure 0 0 0) 0x7f0fd801a070 con 0x7f0fe806fd30 2024-06-09T22:00:46.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.497+0000 7f0fdeffd640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f0fcc06bda0 0x7f0fcc06e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:46.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.498+0000 7f0fee21f640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f0fcc06bda0 0x7f0fcc06e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:46.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.498+0000 7f0fee21f640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f0fcc06bda0 0x7f0fcc06e260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009990 tx=0x7f0fe4009110 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:46.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.503+0000 7f0fdeffd640 1 -- 172.21.15.32:0/4190961955 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0fd8088ea0 con 0x7f0fe806fd30 2024-06-09T22:00:46.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.524+0000 7f0fdeffd640 1 -- 172.21.15.32:0/4190961955 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f0fe80ceac0 con 0x7f0fe806fd30 2024-06-09T22:00:46.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:46.980+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0fe80c34d0 con 0x7f0fe806fd30 2024-06-09T22:00:47.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.229+0000 7f0fdeffd640 1 -- 172.21.15.32:0/4190961955 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f0fd8058b80 con 0x7f0fe806fd30 2024-06-09T22:00:47.213 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:00:47.213 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-06-09T22:00:47.214 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-06-09T22:00:47.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f0fcc06bda0 msgr2=0x7f0fcc06e260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:47.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f0fcc06bda0 0x7f0fcc06e260 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0fe4009990 tx=0x7f0fe4009110 comp rx=0 tx=0).stop 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0fe806fd30 msgr2=0x7f0fe81f3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0fe806fd30 0x7f0fe81f3940 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f0fd800c400 tx=0x7f0fd800c800 comp rx=0 tx=0).stop 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 shutdown_connections 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f0fe80c9730 0x7f0fe81f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f0fcc06bda0 0x7f0fcc06e260 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f0fe806fd30 0x7f0fe81f3940 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 --2- 172.21.15.32:0/4190961955 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0fe806f950 0x7f0fe814f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 >> 172.21.15.32:0/4190961955 conn(0x7f0fe8071680 msgr2=0x7f0fe81593a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 shutdown_connections 2024-06-09T22:00:47.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:47.232+0000 7f0ff0483640 1 -- 172.21.15.32:0/4190961955 wait complete. 2024-06-09T22:00:49.233 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-09T22:00:49.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.367+0000 7f99cc872640 1 Processor -- start 2024-06-09T22:00:49.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.367+0000 7f99cc872640 1 -- start start 2024-06-09T22:00:49.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.368+0000 7f99cc872640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f99c81524b0 0x7f99c80c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:49.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.368+0000 7f99cc872640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99c8152890 0x7f99c80c8880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:49.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.368+0000 7f99cc872640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99c80c2d10 con 0x7f99c80c9590 2024-06-09T22:00:49.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.368+0000 7f99cc872640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f99c8070c10 con 0x7f99c8152890 2024-06-09T22:00:49.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.368+0000 7f99cc872640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f99c80c99c0 con 0x7f99c81524b0 2024-06-09T22:00:49.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.405+0000 7f99c5d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99c8152890 0x7f99c80c8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:49.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.406+0000 7f99c5d9b640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f99c8152890 0x7f99c80c8880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53978/0 (socket says 172.21.15.32:53978) 2024-06-09T22:00:49.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.406+0000 7f99c5d9b640 1 -- 172.21.15.32:0/2555561523 learned_addr learned my addr 172.21.15.32:0/2555561523 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:49.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.409+0000 7f99c5d9b640 1 -- 172.21.15.32:0/2555561523 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f99c81524b0 msgr2=0x7f99c80c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:49.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.409+0000 7f99c5d9b640 1 --2- 172.21.15.32:0/2555561523 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f99c81524b0 0x7f99c80c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:49.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.409+0000 7f99c5d9b640 1 -- 172.21.15.32:0/2555561523 >> v1:172.21.15.32:6789/0 conn(0x7f99c80c9590 legacy=0x7f99c8158e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:49.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.409+0000 7f99c5d9b640 1 -- 172.21.15.32:0/2555561523 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c8070ed0 con 0x7f99c8152890 2024-06-09T22:00:49.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.411+0000 7f99c5d9b640 1 --2- 172.21.15.32:0/2555561523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 0x7f99c80c8880 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009870 tx=0x7f99bc02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4074f39e7a798e62 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:49.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.420+0000 7f99c559a640 1 -- 172.21.15.32:0/2555561523 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc037070 con 0x7f99c8152890 2024-06-09T22:00:49.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.420+0000 7f99c559a640 1 -- 172.21.15.32:0/2555561523 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99bc02ae50 con 0x7f99c8152890 2024-06-09T22:00:49.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.420+0000 7f99c559a640 1 -- 172.21.15.32:0/2555561523 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99bc032860 con 0x7f99c8152890 2024-06-09T22:00:49.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.421+0000 7f99cc872640 1 -- 172.21.15.32:0/2555561523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 msgr2=0x7f99c80c8880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:49.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.421+0000 7f99cc872640 1 --2- 172.21.15.32:0/2555561523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 0x7f99c80c8880 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f99bc009870 tx=0x7f99bc02a300 comp rx=0 tx=0).stop 2024-06-09T22:00:49.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.421+0000 7f99cc872640 1 -- 172.21.15.32:0/2555561523 shutdown_connections 2024-06-09T22:00:49.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.421+0000 7f99cc872640 1 --2- 172.21.15.32:0/2555561523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 0x7f99c80c8880 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:49.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.421+0000 7f99cc872640 1 --2- 172.21.15.32:0/2555561523 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f99c81524b0 0x7f99c80c3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:49.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.421+0000 7f99cc872640 1 -- 172.21.15.32:0/2555561523 >> 172.21.15.32:0/2555561523 conn(0x7f99c8071680 msgr2=0x7f99c8071a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:49.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.421+0000 7f99cc872640 1 -- 172.21.15.32:0/2555561523 shutdown_connections 2024-06-09T22:00:49.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.421+0000 7f99cc872640 1 -- 172.21.15.32:0/2555561523 wait complete. 2024-06-09T22:00:49.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.422+0000 7f99cc872640 1 Processor -- start 2024-06-09T22:00:49.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.422+0000 7f99cc872640 1 -- start start 2024-06-09T22:00:49.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.423+0000 7f99cc872640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80c9590 0x7f99c8150960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:49.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.423+0000 7f99cc872640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f99c81524b0 0x7f99c815a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:49.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.423+0000 7f99cc872640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 0x7f99c815b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:49.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.423+0000 7f99cc872640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99c80c3540 con 0x7f99c8152890 2024-06-09T22:00:49.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.423+0000 7f99cc872640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f99c80c3260 con 0x7f99c80c9590 2024-06-09T22:00:49.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.423+0000 7f99cc872640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f99c80711b0 con 0x7f99c81524b0 2024-06-09T22:00:49.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.424+0000 7f99c659c640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80c9590 0x7f99c8150960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:49.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.424+0000 7f99c6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 0x7f99c815b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:49.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.424+0000 7f99c6d9d640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 0x7f99c815b690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53992/0 (socket says 172.21.15.32:53992) 2024-06-09T22:00:49.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.424+0000 7f99c6d9d640 1 -- 172.21.15.32:0/3144848445 learned_addr learned my addr 172.21.15.32:0/3144848445 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:49.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.428+0000 7f99c659c640 1 -- 172.21.15.32:0/3144848445 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f99c81524b0 msgr2=0x7f99c815a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:49.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.428+0000 7f99c659c640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f99c81524b0 0x7f99c815a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:49.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.428+0000 7f99c659c640 1 -- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 msgr2=0x7f99c815b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:49.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.428+0000 7f99c659c640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 0x7f99c815b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:49.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.428+0000 7f99c659c640 1 -- 172.21.15.32:0/3144848445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99c814cc60 con 0x7f99c80c9590 2024-06-09T22:00:49.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.431+0000 7f99c659c640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80c9590 0x7f99c8150960 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99b800b420 tx=0x7f99b800b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:49.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.435+0000 7f99b6ffd640 1 -- 172.21.15.32:0/3144848445 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8011830 con 0x7f99c80c9590 2024-06-09T22:00:49.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.435+0000 7f99b6ffd640 1 -- 172.21.15.32:0/3144848445 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99b8016070 con 0x7f99c80c9590 2024-06-09T22:00:49.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.435+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99c81597a0 con 0x7f99c80c9590 2024-06-09T22:00:49.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.435+0000 7f99b6ffd640 1 -- 172.21.15.32:0/3144848445 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99b8011c10 con 0x7f99c80c9590 2024-06-09T22:00:49.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.435+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99c814e670 con 0x7f99c80c9590 2024-06-09T22:00:49.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.436+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99c8068490 con 0x7f99c80c9590 2024-06-09T22:00:49.677 INFO:tasks.ceph.mds.b.smithi148.stdout:starting mds.b at 2024-06-09T22:00:49.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.764+0000 7f99b6ffd640 1 -- 172.21.15.32:0/3144848445 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f99b801b460 con 0x7f99c80c9590 2024-06-09T22:00:49.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.765+0000 7f99b6ffd640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f99a806be80 0x7f99a806e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:49.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.765+0000 7f99c5d9b640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f99a806be80 0x7f99a806e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:49.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.766+0000 7f99c5d9b640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f99a806be80 0x7f99a806e340 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f99bc034000 tx=0x7f99bc02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:49.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.767+0000 7f99b6ffd640 1 -- 172.21.15.32:0/3144848445 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f99b80600d0 con 0x7f99c80c9590 2024-06-09T22:00:49.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:49.792+0000 7f99b6ffd640 1 -- 172.21.15.32:0/3144848445 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f99b8059f30 con 0x7f99c80c9590 2024-06-09T22:00:49.792 INFO:tasks.ceph.mds.f.smithi148.stdout:starting mds.f at 2024-06-09T22:00:49.875 INFO:tasks.ceph.mds.d.smithi148.stdout:starting mds.d at 2024-06-09T22:00:50.032 INFO:tasks.ceph.mds.a.smithi032.stdout:starting mds.a at 2024-06-09T22:00:50.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.064+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f99c81501d0 con 0x7f99c80c9590 2024-06-09T22:00:50.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.073+0000 7f99b6ffd640 1 -- 172.21.15.32:0/3144848445 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f99b8059830 con 0x7f99c80c9590 2024-06-09T22:00:50.058 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:00:50.058 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-06-09T22:00:50.058 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-06-09T22:00:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f99a806be80 msgr2=0x7f99a806e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f99a806be80 0x7f99a806e340 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f99bc034000 tx=0x7f99bc02c000 comp rx=0 tx=0).stop 2024-06-09T22:00:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80c9590 msgr2=0x7f99c8150960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:50.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80c9590 0x7f99c8150960 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f99b800b420 tx=0x7f99b800b8e0 comp rx=0 tx=0).stop 2024-06-09T22:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 shutdown_connections 2024-06-09T22:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99c8152890 0x7f99c815b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f99a806be80 0x7f99a806e340 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f99c81524b0 0x7f99c815a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 --2- 172.21.15.32:0/3144848445 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f99c80c9590 0x7f99c8150960 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 >> 172.21.15.32:0/3144848445 conn(0x7f99c8071680 msgr2=0x7f99c80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 shutdown_connections 2024-06-09T22:00:50.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:50.076+0000 7f99cc872640 1 -- 172.21.15.32:0/3144848445 wait complete. 2024-06-09T22:00:50.099 INFO:tasks.ceph.mds.e.smithi032.stdout:starting mds.e at 2024-06-09T22:00:50.133 INFO:tasks.ceph.mds.c.smithi032.stdout:starting mds.c at 2024-06-09T22:00:52.073 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-09T22:00:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.197+0000 7f18b6838640 1 Processor -- start 2024-06-09T22:00:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.197+0000 7f18b6838640 1 -- start start 2024-06-09T22:00:52.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.198+0000 7f18b6838640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b006f310 0x7f18b006b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.198+0000 7f18b6838640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b00703e0 con 0x7f18b006f9e0 2024-06-09T22:00:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.198+0000 7f18b6838640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18b006e8b0 con 0x7f18b01525f0 2024-06-09T22:00:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.198+0000 7f18b6838640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f18b00706e0 con 0x7f18b006f310 2024-06-09T22:00:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.199+0000 7f18b4dd5640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f18b006f9e0 0x7f18b0162c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51800/0 (socket says 172.21.15.32:51800) 2024-06-09T22:00:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.199+0000 7f18b4dd5640 1 -- 172.21.15.32:0/2652546225 learned_addr learned my addr 172.21.15.32:0/2652546225 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:00:52.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.199+0000 7f18af7fe640 1 --2- 172.21.15.32:0/2652546225 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b006f310 0x7f18b006b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.202+0000 7f18aeffd640 1 -- 172.21.15.32:0/2652546225 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1757903341 0 0) 0x7f18b00703e0 con 0x7f18b006f9e0 2024-06-09T22:00:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.202+0000 7f18aeffd640 1 -- 172.21.15.32:0/2652546225 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1898003520 con 0x7f18b006f9e0 2024-06-09T22:00:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.202+0000 7f18aeffd640 1 -- 172.21.15.32:0/2652546225 <== mon.1 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4290861237 0 0) 0x7f18b006e8b0 con 0x7f18b01525f0 2024-06-09T22:00:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.202+0000 7f18aeffd640 1 -- 172.21.15.32:0/2652546225 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18b00703e0 con 0x7f18b01525f0 2024-06-09T22:00:52.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.202+0000 7f18af7fe640 1 -- 172.21.15.32:0/2652546225 >> v1:172.21.15.148:6789/0 conn(0x7f18b01525f0 legacy=0x7f18b0070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.202+0000 7f18af7fe640 1 -- 172.21.15.32:0/2652546225 >> v1:172.21.15.32:6789/0 conn(0x7f18b006f9e0 legacy=0x7f18b0162c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.202+0000 7f18af7fe640 1 -- 172.21.15.32:0/2652546225 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b01529d0 con 0x7f18b006f310 2024-06-09T22:00:52.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.205+0000 7f18af7fe640 1 --2- 172.21.15.32:0/2652546225 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b006f310 0x7f18b006b390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18a40097d0 tx=0x7f18a402a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4579c38e64457ee8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.208+0000 7f18aeffd640 1 -- 172.21.15.32:0/2652546225 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4037070 con 0x7f18b006f310 2024-06-09T22:00:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18aeffd640 1 -- 172.21.15.32:0/2652546225 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a4032440 con 0x7f18b006f310 2024-06-09T22:00:52.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18aeffd640 1 -- 172.21.15.32:0/2652546225 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a4032810 con 0x7f18b006f310 2024-06-09T22:00:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18b6838640 1 -- 172.21.15.32:0/2652546225 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b006f310 msgr2=0x7f18b006b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18b6838640 1 --2- 172.21.15.32:0/2652546225 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b006f310 0x7f18b006b390 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f18a40097d0 tx=0x7f18a402a180 comp rx=0 tx=0).stop 2024-06-09T22:00:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18b6838640 1 -- 172.21.15.32:0/2652546225 shutdown_connections 2024-06-09T22:00:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18b6838640 1 --2- 172.21.15.32:0/2652546225 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b006f310 0x7f18b006b390 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18b6838640 1 -- 172.21.15.32:0/2652546225 >> 172.21.15.32:0/2652546225 conn(0x7f18b00c9930 msgr2=0x7f18b00c9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18b6838640 1 -- 172.21.15.32:0/2652546225 shutdown_connections 2024-06-09T22:00:52.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.209+0000 7f18b6838640 1 -- 172.21.15.32:0/2652546225 wait complete. 2024-06-09T22:00:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.210+0000 7f18b6838640 1 Processor -- start 2024-06-09T22:00:52.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.211+0000 7f18b6838640 1 -- start start 2024-06-09T22:00:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.211+0000 7f18b6838640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18b006f310 0x7f18b01e5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.211+0000 7f18b6838640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18b006f9e0 0x7f18b00c36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.211+0000 7f18b6838640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b01525f0 0x7f18b00b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.211+0000 7f18b6838640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f18b006ee50 con 0x7f18b006f310 2024-06-09T22:00:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.211+0000 7f18b6838640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f18b006eb70 con 0x7f18b006f9e0 2024-06-09T22:00:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.211+0000 7f18b6838640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f18b0071280 con 0x7f18b01525f0 2024-06-09T22:00:52.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.212+0000 7f18affff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18b006f310 0x7f18b01e5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.212+0000 7f18af7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18b006f9e0 0x7f18b00c36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.212+0000 7f18b4dd5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b01525f0 0x7f18b00b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.212+0000 7f18affff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18b006f310 0x7f18b01e5120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55406/0 (socket says 172.21.15.32:55406) 2024-06-09T22:00:52.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.212+0000 7f18affff640 1 -- 172.21.15.32:0/3627329332 learned_addr learned my addr 172.21.15.32:0/3627329332 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.216+0000 7f18affff640 1 -- 172.21.15.32:0/3627329332 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b01525f0 msgr2=0x7f18b00b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.216+0000 7f18affff640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b01525f0 0x7f18b00b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.216+0000 7f18affff640 1 -- 172.21.15.32:0/3627329332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18b006f9e0 msgr2=0x7f18b00c36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.216+0000 7f18affff640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18b006f9e0 0x7f18b00c36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.216+0000 7f18affff640 1 -- 172.21.15.32:0/3627329332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18b01f0240 con 0x7f18b006f310 2024-06-09T22:00:52.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.218+0000 7f18affff640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18b006f310 0x7f18b01e5120 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f18a000c460 tx=0x7f18a000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:52.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.222+0000 7f18acff9640 1 -- 172.21.15.32:0/3627329332 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a00052d0 con 0x7f18b006f310 2024-06-09T22:00:52.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.222+0000 7f18acff9640 1 -- 172.21.15.32:0/3627329332 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a0005490 con 0x7f18b006f310 2024-06-09T22:00:52.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.222+0000 7f18acff9640 1 -- 172.21.15.32:0/3627329332 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a0012870 con 0x7f18b006f310 2024-06-09T22:00:52.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.222+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18b01f0520 con 0x7f18b006f310 2024-06-09T22:00:52.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.222+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18b01f0ae0 con 0x7f18b006f310 2024-06-09T22:00:52.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.223+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18b0068490 con 0x7f18b006f310 2024-06-09T22:00:52.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.243+0000 7f18acff9640 1 -- 172.21.15.32:0/3627329332 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f18a0012a10 con 0x7f18b006f310 2024-06-09T22:00:52.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.244+0000 7f18acff9640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f189c06be80 0x7f189c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.244+0000 7f18af7fe640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f189c06be80 0x7f189c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.244+0000 7f18af7fe640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f189c06be80 0x7f189c06e340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f18a4034000 tx=0x7f18a402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:52.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.247+0000 7f18acff9640 1 -- 172.21.15.32:0/3627329332 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f18a005ff60 con 0x7f18b006f310 2024-06-09T22:00:52.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.527+0000 7f18acff9640 1 -- 172.21.15.32:0/3627329332 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f18a0059770 con 0x7f18b006f310 2024-06-09T22:00:52.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.770+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f18b006d4b0 con 0x7f18b006f310 2024-06-09T22:00:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.796+0000 7f18acff9640 1 -- 172.21.15.32:0/3627329332 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5521 (secure 0 0 0) 0x7f18a005da70 con 0x7f18b006f310 2024-06-09T22:00:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:00:52.780 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4199,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2870873470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2870873470},{"type":"v1","addr":"172.21.15.32:6835","nonce":2870873470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/1513914407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":1513914407},{"type":"v1","addr":"172.21.15.148:6833","nonce":1513914407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4243,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2699555510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":2699555510},{"type":"v1","addr":"172.21.15.148:6835","nonce":2699555510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2047518031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2047518031},{"type":"v1","addr":"172.21.15.148:6837","nonce":2047518031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2947563635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2947563635},{"type":"v1","addr":"172.21.15.32:6837","nonce":2947563635}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4263,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/3734386334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":3734386334},{"type":"v1","addr":"172.21.15.32:6839","nonce":3734386334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-06-09T22:00:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 3 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f189c06be80 msgr2=0x7f189c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f189c06be80 0x7f189c06e340 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f18a4034000 tx=0x7f18a402e000 comp rx=0 tx=0).stop 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18b006f310 msgr2=0x7f18b01e5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18b006f310 0x7f18b01e5120 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f18a000c460 tx=0x7f18a000c920 comp rx=0 tx=0).stop 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 shutdown_connections 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f189c06be80 0x7f189c06e340 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f18b01525f0 0x7f18b00b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f18b006f9e0 0x7f18b00c36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 --2- 172.21.15.32:0/3627329332 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f18b006f310 0x7f18b01e5120 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.798+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 >> 172.21.15.32:0/3627329332 conn(0x7f18b00c9930 msgr2=0x7f18b00cad30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.799+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 shutdown_connections 2024-06-09T22:00:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.799+0000 7f18b6838640 1 -- 172.21.15.32:0/3627329332 wait complete. 2024-06-09T22:00:52.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-06-09T22:00:52.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.913+0000 7fb077189640 1 Processor -- start 2024-06-09T22:00:52.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.913+0000 7fb077189640 1 -- start start 2024-06-09T22:00:52.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.914+0000 7fb077189640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fb0701549c0 0x7fb0700ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.914+0000 7fb077189640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb07015e5f0 0x7fb0700c8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.914+0000 7fb077189640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb070154da0 con 0x7fb0700c91a0 2024-06-09T22:00:52.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.914+0000 7fb077189640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7fb0700cdf80 con 0x7fb0701549c0 2024-06-09T22:00:52.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.914+0000 7fb077189640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fb0700c4200 con 0x7fb07015e5f0 2024-06-09T22:00:52.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.915+0000 7fb075726640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7fb0700c91a0 0x7fb0700ca820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.32:47418/0 (socket says 172.21.15.32:47418) 2024-06-09T22:00:52.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.915+0000 7fb075726640 1 -- 172.21.15.32:0/3131166519 learned_addr learned my addr 172.21.15.32:0/3131166519 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:00:52.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.915+0000 7fb074f25640 1 --2- 172.21.15.32:0/3131166519 >> v2:172.21.15.32:3300/0 conn(0x7fb0701549c0 0x7fb0700ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.917+0000 7fb06f7fe640 1 -- 172.21.15.32:0/3131166519 <== mon.1 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1857522961 0 0) 0x7fb070154da0 con 0x7fb0700c91a0 2024-06-09T22:00:52.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.917+0000 7fb06f7fe640 1 -- 172.21.15.32:0/3131166519 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb060003520 con 0x7fb0700c91a0 2024-06-09T22:00:52.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.918+0000 7fb074f25640 1 -- 172.21.15.32:0/3131166519 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb07015e5f0 msgr2=0x7fb0700c8c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:52.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.918+0000 7fb074f25640 1 --2- 172.21.15.32:0/3131166519 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb07015e5f0 0x7fb0700c8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.918+0000 7fb074f25640 1 -- 172.21.15.32:0/3131166519 >> v1:172.21.15.148:6789/0 conn(0x7fb0700c91a0 legacy=0x7fb0700ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.918+0000 7fb074f25640 1 -- 172.21.15.32:0/3131166519 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0700c44e0 con 0x7fb0701549c0 2024-06-09T22:00:52.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.920+0000 7fb074f25640 1 --2- 172.21.15.32:0/3131166519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0701549c0 0x7fb0700ce2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb0640097a0 tx=0x7fb06402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a06f154b380ae55 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.923+0000 7fb06f7fe640 1 -- 172.21.15.32:0/3131166519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb064037070 con 0x7fb0701549c0 2024-06-09T22:00:52.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb06f7fe640 1 -- 172.21.15.32:0/3131166519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb064032440 con 0x7fb0701549c0 2024-06-09T22:00:52.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb06f7fe640 1 -- 172.21.15.32:0/3131166519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb064032810 con 0x7fb0701549c0 2024-06-09T22:00:52.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb077189640 1 -- 172.21.15.32:0/3131166519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0701549c0 msgr2=0x7fb0700ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb077189640 1 --2- 172.21.15.32:0/3131166519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0701549c0 0x7fb0700ce2d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb0640097a0 tx=0x7fb06402a210 comp rx=0 tx=0).stop 2024-06-09T22:00:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb077189640 1 -- 172.21.15.32:0/3131166519 shutdown_connections 2024-06-09T22:00:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb077189640 1 --2- 172.21.15.32:0/3131166519 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb07015e5f0 0x7fb0700c8c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb077189640 1 --2- 172.21.15.32:0/3131166519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0701549c0 0x7fb0700ce2d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb077189640 1 -- 172.21.15.32:0/3131166519 >> 172.21.15.32:0/3131166519 conn(0x7fb070071750 msgr2=0x7fb070071b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb077189640 1 -- 172.21.15.32:0/3131166519 shutdown_connections 2024-06-09T22:00:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.924+0000 7fb077189640 1 -- 172.21.15.32:0/3131166519 wait complete. 2024-06-09T22:00:52.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.926+0000 7fb077189640 1 Processor -- start 2024-06-09T22:00:52.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.926+0000 7fb077189640 1 -- start start 2024-06-09T22:00:52.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.926+0000 7fb077189640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0700c91a0 0x7fb0701f4290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.926+0000 7fb077189640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0701549c0 0x7fb070151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.926+0000 7fb077189640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb07015e5f0 0x7fb0700c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.926+0000 7fb077189640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0700c4d80 con 0x7fb0700c91a0 2024-06-09T22:00:52.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.926+0000 7fb077189640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fb0700c4aa0 con 0x7fb0701549c0 2024-06-09T22:00:52.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.926+0000 7fb077189640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fb0700ca300 con 0x7fb07015e5f0 2024-06-09T22:00:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.927+0000 7fb074f25640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0700c91a0 0x7fb0701f4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.927+0000 7fb06ffff640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0701549c0 0x7fb070151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.927+0000 7fb074f25640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0700c91a0 0x7fb0701f4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55420/0 (socket says 172.21.15.32:55420) 2024-06-09T22:00:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.927+0000 7fb074f25640 1 -- 172.21.15.32:0/2985975198 learned_addr learned my addr 172.21.15.32:0/2985975198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:52.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.931+0000 7fb074f25640 1 -- 172.21.15.32:0/2985975198 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb07015e5f0 msgr2=0x7fb0700c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.931+0000 7fb074f25640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb07015e5f0 0x7fb0700c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.931+0000 7fb074f25640 1 -- 172.21.15.32:0/2985975198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0701549c0 msgr2=0x7fb070151ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:52.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.931+0000 7fb074f25640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0701549c0 0x7fb070151ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:52.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.931+0000 7fb074f25640 1 -- 172.21.15.32:0/2985975198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0700c70d0 con 0x7fb0700c91a0 2024-06-09T22:00:52.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.933+0000 7fb074f25640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0700c91a0 0x7fb0701f4290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb06402a800 tx=0x7fb0640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:52.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.953+0000 7fb06d7fa640 1 -- 172.21.15.32:0/2985975198 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb064037050 con 0x7fb0700c91a0 2024-06-09T22:00:52.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.953+0000 7fb06d7fa640 1 -- 172.21.15.32:0/2985975198 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb064038040 con 0x7fb0700c91a0 2024-06-09T22:00:52.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.953+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0700c7bc0 con 0x7fb0700c91a0 2024-06-09T22:00:52.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.953+0000 7fb06d7fa640 1 -- 172.21.15.32:0/2985975198 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb06403cb70 con 0x7fb0700c91a0 2024-06-09T22:00:52.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.953+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0700c60b0 con 0x7fb0700c91a0 2024-06-09T22:00:52.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.954+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb070074f90 con 0x7fb0700c91a0 2024-06-09T22:00:52.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.975+0000 7fb06d7fa640 1 -- 172.21.15.32:0/2985975198 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fb06403cd10 con 0x7fb0700c91a0 2024-06-09T22:00:52.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.975+0000 7fb06d7fa640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fb05406be80 0x7fb05406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.976+0000 7fb06ffff640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fb05406be80 0x7fb05406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:52.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:52.976+0000 7fb06ffff640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fb05406be80 0x7fb05406e340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb070071e80 tx=0x7fb05c009060 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:53.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.256+0000 7fb06d7fa640 1 -- 172.21.15.32:0/2985975198 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fb064078640 con 0x7fb0700c91a0 2024-06-09T22:00:53.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.256+0000 7fb06d7fa640 1 -- 172.21.15.32:0/2985975198 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7fb06407aca0 con 0x7fb0700c91a0 2024-06-09T22:00:53.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.458+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fb07015b5b0 con 0x7fb0700c91a0 2024-06-09T22:00:53.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.467+0000 7fb06d7fa640 1 -- 172.21.15.32:0/2985975198 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fb06407a5a0 con 0x7fb0700c91a0 2024-06-09T22:00:53.451 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:00:53.451 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-06-09T22:00:53.451 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-06-09T22:00:53.452 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-06-09T22:00:53.452 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-06-09T22:00:53.452 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-06-09T22:00:53.452 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fb05406be80 msgr2=0x7fb05406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fb05406be80 0x7fb05406e340 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb070071e80 tx=0x7fb05c009060 comp rx=0 tx=0).stop 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0700c91a0 msgr2=0x7fb0701f4290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0700c91a0 0x7fb0701f4290 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb06402a800 tx=0x7fb0640046c0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 shutdown_connections 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fb05406be80 0x7fb05406e340 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fb07015e5f0 0x7fb0700c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fb0701549c0 0x7fb070151ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.469+0000 7fb077189640 1 --2- 172.21.15.32:0/2985975198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0700c91a0 0x7fb0701f4290 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.470+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 >> 172.21.15.32:0/2985975198 conn(0x7fb070071750 msgr2=0x7fb07015d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:53.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.470+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 shutdown_connections 2024-06-09T22:00:53.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.470+0000 7fb077189640 1 -- 172.21.15.32:0/2985975198 wait complete. 2024-06-09T22:00:53.464 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-06-09T22:00:53.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-06-09T22:00:53.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.590+0000 7f8960529640 1 Processor -- start 2024-06-09T22:00:53.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.590+0000 7f8960529640 1 -- start start 2024-06-09T22:00:53.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.591+0000 7f8960529640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:53.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.592+0000 7f8960529640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89580cec00 0x7f8958164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:53.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.592+0000 7f8960529640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89580cd710 con 0x7f89580ce6b0 2024-06-09T22:00:53.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.592+0000 7f8960529640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7f89580ca6e0 con 0x7f89580cec00 2024-06-09T22:00:53.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.592+0000 7f8960529640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f895815e030 con 0x7f8958154b30 2024-06-09T22:00:53.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.608+0000 7f895e2c5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:53.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.608+0000 7f895eac6640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f89580cec00 0x7f8958164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:53.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.608+0000 7f895e2c5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.32:35894/0 (socket says 172.21.15.32:35894) 2024-06-09T22:00:53.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.608+0000 7f895e2c5640 1 -- 172.21.15.32:0/3577528833 learned_addr learned my addr 172.21.15.32:0/3577528833 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:53.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.612+0000 7f895e2c5640 1 -- 172.21.15.32:0/3577528833 >> v2:172.21.15.148:3300/0 conn(0x7f89580cec00 msgr2=0x7f8958164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:53.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.612+0000 7f895e2c5640 1 --2- 172.21.15.32:0/3577528833 >> v2:172.21.15.148:3300/0 conn(0x7f89580cec00 0x7f8958164f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.612+0000 7f895e2c5640 1 -- 172.21.15.32:0/3577528833 >> v1:172.21.15.148:6789/0 conn(0x7f89580ce6b0 legacy=0x7f89580cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:53.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.612+0000 7f895e2c5640 1 -- 172.21.15.32:0/3577528833 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f895815dd00 con 0x7f8958154b30 2024-06-09T22:00:53.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.618+0000 7f895e2c5640 1 --2- 172.21.15.32:0/3577528833 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815e4c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f894c009870 tx=0x7f894c02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=acd39c3a3e1f953b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.622+0000 7f895d2c3640 1 -- 172.21.15.32:0/3577528833 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c037070 con 0x7f8958154b30 2024-06-09T22:00:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f895d2c3640 1 -- 172.21.15.32:0/3577528833 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f894c032440 con 0x7f8958154b30 2024-06-09T22:00:53.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f895d2c3640 1 -- 172.21.15.32:0/3577528833 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f894c032810 con 0x7f8958154b30 2024-06-09T22:00:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f8960529640 1 -- 172.21.15.32:0/3577528833 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 msgr2=0x7f895815e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f8960529640 1 --2- 172.21.15.32:0/3577528833 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815e4c0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f894c009870 tx=0x7f894c02a1c0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f8960529640 1 -- 172.21.15.32:0/3577528833 shutdown_connections 2024-06-09T22:00:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f8960529640 1 --2- 172.21.15.32:0/3577528833 >> v2:172.21.15.148:3300/0 conn(0x7f89580cec00 0x7f8958164f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f8960529640 1 --2- 172.21.15.32:0/3577528833 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815e4c0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f8960529640 1 -- 172.21.15.32:0/3577528833 >> 172.21.15.32:0/3577528833 conn(0x7f89580b1870 msgr2=0x7f89580b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f8960529640 1 -- 172.21.15.32:0/3577528833 shutdown_connections 2024-06-09T22:00:53.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.623+0000 7f8960529640 1 -- 172.21.15.32:0/3577528833 wait complete. 2024-06-09T22:00:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.625+0000 7f8960529640 1 Processor -- start 2024-06-09T22:00:53.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.625+0000 7f8960529640 1 -- start start 2024-06-09T22:00:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.625+0000 7f8960529640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89580ce6b0 0x7f89581f0b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.625+0000 7f8960529640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89580cec00 0x7f89580c5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.625+0000 7f8960529640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.625+0000 7f8960529640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f89580caef0 con 0x7f89580ce6b0 2024-06-09T22:00:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.625+0000 7f8960529640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f89580cac10 con 0x7f89580cec00 2024-06-09T22:00:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.625+0000 7f8960529640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f895815ea00 con 0x7f8958154b30 2024-06-09T22:00:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.626+0000 7f895eac6640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:53.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.626+0000 7f895dac4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89580cec00 0x7f89580c5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.627+0000 7f895eac6640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.32:35906/0 (socket says 172.21.15.32:35906) 2024-06-09T22:00:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.627+0000 7f895dac4640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89580cec00 0x7f89580c5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:45508/0 (socket says 172.21.15.32:45508) 2024-06-09T22:00:53.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.627+0000 7f895eac6640 1 -- 172.21.15.32:0/4171512133 learned_addr learned my addr 172.21.15.32:0/4171512133 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:53.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.631+0000 7f895eac6640 1 -- 172.21.15.32:0/4171512133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89580cec00 msgr2=0x7f89580c5b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:53.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.631+0000 7f895eac6640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89580cec00 0x7f89580c5b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.631+0000 7f895eac6640 1 -- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89580ce6b0 msgr2=0x7f89581f0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:53.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.631+0000 7f895eac6640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89580ce6b0 0x7f89581f0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:53.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.631+0000 7f895eac6640 1 -- 172.21.15.32:0/4171512133 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89580c4920 con 0x7f8958154b30 2024-06-09T22:00:53.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.634+0000 7f895eac6640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815c400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8950009e30 tx=0x7f895000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:53.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.638+0000 7f894affd640 1 -- 172.21.15.32:0/4171512133 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950018020 con 0x7f8958154b30 2024-06-09T22:00:53.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.638+0000 7f894affd640 1 -- 172.21.15.32:0/4171512133 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8950014480 con 0x7f8958154b30 2024-06-09T22:00:53.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.638+0000 7f894affd640 1 -- 172.21.15.32:0/4171512133 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8950014870 con 0x7f8958154b30 2024-06-09T22:00:53.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.638+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89580c6400 con 0x7f8958154b30 2024-06-09T22:00:53.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.638+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f89580c4ea0 con 0x7f8958154b30 2024-06-09T22:00:53.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.639+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8958068490 con 0x7f8958154b30 2024-06-09T22:00:53.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.657+0000 7f894affd640 1 -- 172.21.15.32:0/4171512133 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f895001b020 con 0x7f8958154b30 2024-06-09T22:00:53.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.658+0000 7f894affd640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f893c06be80 0x7f893c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:53.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.658+0000 7f895e2c5640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f893c06be80 0x7f893c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:53.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.658+0000 7f895e2c5640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f893c06be80 0x7f893c06e340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f894c034000 tx=0x7f894c02c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:53.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.660+0000 7f894affd640 1 -- 172.21.15.32:0/4171512133 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f8950058680 con 0x7f8958154b30 2024-06-09T22:00:53.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:53.905+0000 7f894affd640 1 -- 172.21.15.32:0/4171512133 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f8950022020 con 0x7f8958154b30 2024-06-09T22:00:54.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.088+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f895806e110 con 0x7f8958154b30 2024-06-09T22:00:54.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.815+0000 7f894affd640 1 -- 172.21.15.32:0/4171512133 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7f8950010040 con 0x7f8958154b30 2024-06-09T22:00:54.800 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-06-09T22:00:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.817+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f893c06be80 msgr2=0x7f893c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.817+0000 7f8960529640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f893c06be80 0x7f893c06e340 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f894c034000 tx=0x7f894c02c000 comp rx=0 tx=0).stop 2024-06-09T22:00:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.817+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 msgr2=0x7f895815c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.817+0000 7f8960529640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815c400 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8950009e30 tx=0x7f895000c490 comp rx=0 tx=0).stop 2024-06-09T22:00:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.818+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 shutdown_connections 2024-06-09T22:00:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.818+0000 7f8960529640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f893c06be80 0x7f893c06e340 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:54.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.818+0000 7f8960529640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f8958154b30 0x7f895815c400 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:54.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.818+0000 7f8960529640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f89580cec00 0x7f89580c5b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:54.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.818+0000 7f8960529640 1 --2- 172.21.15.32:0/4171512133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f89580ce6b0 0x7f89581f0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:54.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.818+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 >> 172.21.15.32:0/4171512133 conn(0x7f89580b1870 msgr2=0x7f89580cf0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:54.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.818+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 shutdown_connections 2024-06-09T22:00:54.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.818+0000 7f8960529640 1 -- 172.21.15.32:0/4171512133 wait complete. 2024-06-09T22:00:54.815 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-06-09T22:00:54.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.946+0000 7f1e647ed640 1 Processor -- start 2024-06-09T22:00:54.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.946+0000 7f1e647ed640 1 -- start start 2024-06-09T22:00:54.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.947+0000 7f1e647ed640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1e5c0ce7a0 0x7f1e5c0cf680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:54.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.947+0000 7f1e647ed640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e5c15a9f0 con 0x7f1e5c0cf230 2024-06-09T22:00:54.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.947+0000 7f1e647ed640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e5c15acb0 con 0x7f1e5c0ccbc0 2024-06-09T22:00:54.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.947+0000 7f1e647ed640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7f1e5c15af90 con 0x7f1e5c0ce7a0 2024-06-09T22:00:54.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.947+0000 7f1e62d8a640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f1e5c0ce7a0 0x7f1e5c0cf680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:54.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.948+0000 7f1e62589640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f1e5c0ccbc0 0x7f1e5c0ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.32:47430/0 (socket says 172.21.15.32:47430) 2024-06-09T22:00:54.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.948+0000 7f1e62589640 1 -- 172.21.15.32:0/735241475 learned_addr learned my addr 172.21.15.32:0/735241475 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:00:54.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.950+0000 7f1e62d8a640 1 -- 172.21.15.32:0/735241475 >> v1:172.21.15.148:6789/0 conn(0x7f1e5c0ccbc0 legacy=0x7f1e5c0ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:54.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.950+0000 7f1e62d8a640 1 -- 172.21.15.32:0/735241475 >> v1:172.21.15.32:6789/0 conn(0x7f1e5c0cf230 legacy=0x7f1e5c0ce090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:54.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.950+0000 7f1e62d8a640 1 -- 172.21.15.32:0/735241475 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c15b270 con 0x7f1e5c0ce7a0 2024-06-09T22:00:54.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.954+0000 7f1e62d8a640 1 --2- 172.21.15.32:0/735241475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0ce7a0 0x7f1e5c0cf680 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009d90 tx=0x7f1e5402a730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9e3bc21e51daac server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:54.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.958+0000 7f1e61587640 1 -- 172.21.15.32:0/735241475 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54007740 con 0x7f1e5c0ce7a0 2024-06-09T22:00:54.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.958+0000 7f1e61587640 1 -- 172.21.15.32:0/735241475 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e540078e0 con 0x7f1e5c0ce7a0 2024-06-09T22:00:54.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.958+0000 7f1e61587640 1 -- 172.21.15.32:0/735241475 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54007cb0 con 0x7f1e5c0ce7a0 2024-06-09T22:00:54.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.959+0000 7f1e647ed640 1 -- 172.21.15.32:0/735241475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0ce7a0 msgr2=0x7f1e5c0cf680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:54.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.959+0000 7f1e647ed640 1 --2- 172.21.15.32:0/735241475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0ce7a0 0x7f1e5c0cf680 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009d90 tx=0x7f1e5402a730 comp rx=0 tx=0).stop 2024-06-09T22:00:54.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.959+0000 7f1e647ed640 1 -- 172.21.15.32:0/735241475 shutdown_connections 2024-06-09T22:00:54.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.959+0000 7f1e647ed640 1 --2- 172.21.15.32:0/735241475 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0ce7a0 0x7f1e5c0cf680 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:54.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.959+0000 7f1e647ed640 1 -- 172.21.15.32:0/735241475 >> 172.21.15.32:0/735241475 conn(0x7f1e5c0b1880 msgr2=0x7f1e5c0b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:54.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.959+0000 7f1e647ed640 1 -- 172.21.15.32:0/735241475 shutdown_connections 2024-06-09T22:00:54.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.959+0000 7f1e647ed640 1 -- 172.21.15.32:0/735241475 wait complete. 2024-06-09T22:00:54.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.960+0000 7f1e647ed640 1 Processor -- start 2024-06-09T22:00:54.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.960+0000 7f1e647ed640 1 -- start start 2024-06-09T22:00:54.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.961+0000 7f1e647ed640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1e5c0ccbc0 0x7f1e5c151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:54.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.961+0000 7f1e647ed640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5c0ce7a0 0x7f1e5c1523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:54.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.961+0000 7f1e647ed640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0cf230 0x7f1e5c1f0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:54.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.961+0000 7f1e647ed640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c15a6e0 con 0x7f1e5c0ce7a0 2024-06-09T22:00:54.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.961+0000 7f1e647ed640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5c0cdb70 con 0x7f1e5c0cf230 2024-06-09T22:00:54.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.961+0000 7f1e647ed640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f1e5c0cae00 con 0x7f1e5c0ccbc0 2024-06-09T22:00:54.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.962+0000 7f1e62d8a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0cf230 0x7f1e5c1f0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:54.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.962+0000 7f1e61d88640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5c0ce7a0 0x7f1e5c1523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:54.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.962+0000 7f1e62589640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1e5c0ccbc0 0x7f1e5c151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:54.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.962+0000 7f1e61d88640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5c0ce7a0 0x7f1e5c1523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55434/0 (socket says 172.21.15.32:55434) 2024-06-09T22:00:54.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.962+0000 7f1e61d88640 1 -- 172.21.15.32:0/2308318438 learned_addr learned my addr 172.21.15.32:0/2308318438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.965+0000 7f1e62d8a640 1 -- 172.21.15.32:0/2308318438 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1e5c0ccbc0 msgr2=0x7f1e5c151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.965+0000 7f1e62d8a640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1e5c0ccbc0 0x7f1e5c151e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.965+0000 7f1e62d8a640 1 -- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5c0ce7a0 msgr2=0x7f1e5c1523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.965+0000 7f1e62d8a640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5c0ce7a0 0x7f1e5c1523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:54.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.965+0000 7f1e62d8a640 1 -- 172.21.15.32:0/2308318438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5c15d8c0 con 0x7f1e5c0cf230 2024-06-09T22:00:54.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.968+0000 7f1e62d8a640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0cf230 0x7f1e5c1f0690 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e54031000 tx=0x7f1e54004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:54.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.971+0000 7f1e52ffd640 1 -- 172.21.15.32:0/2308318438 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54040070 con 0x7f1e5c0cf230 2024-06-09T22:00:54.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.971+0000 7f1e52ffd640 1 -- 172.21.15.32:0/2308318438 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e54031c50 con 0x7f1e5c0cf230 2024-06-09T22:00:54.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.971+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5c1f3b80 con 0x7f1e5c0cf230 2024-06-09T22:00:54.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.971+0000 7f1e52ffd640 1 -- 172.21.15.32:0/2308318438 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54045460 con 0x7f1e5c0cf230 2024-06-09T22:00:54.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.972+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5c1f4140 con 0x7f1e5c0cf230 2024-06-09T22:00:54.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.972+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e5c068490 con 0x7f1e5c0cf230 2024-06-09T22:00:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.991+0000 7f1e52ffd640 1 -- 172.21.15.32:0/2308318438 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f1e54039040 con 0x7f1e5c0cf230 2024-06-09T22:00:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.991+0000 7f1e52ffd640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f1e4006be80 0x7f1e4006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:54.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.992+0000 7f1e62589640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f1e4006be80 0x7f1e4006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:54.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:54.992+0000 7f1e62589640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f1e4006be80 0x7f1e4006e340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1e58005880 tx=0x7f1e58009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:55.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:55.238+0000 7f1e52ffd640 1 -- 172.21.15.32:0/2308318438 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7f1e54079000 con 0x7f1e5c0cf230 2024-06-09T22:00:55.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:55.249+0000 7f1e52ffd640 1 -- 172.21.15.32:0/2308318438 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f1e54054080 con 0x7f1e5c0cf230 2024-06-09T22:00:55.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:55.439+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f1e5c06e270 con 0x7f1e5c0cf230 2024-06-09T22:00:56.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.016+0000 7f1e52ffd640 1 -- 172.21.15.32:0/2308318438 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v19) ==== 171+0+0 (secure 0 0 0) 0x7f1e5407af60 con 0x7f1e5c0cf230 2024-06-09T22:00:56.001 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-06-09T22:00:56.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.018+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f1e4006be80 msgr2=0x7f1e4006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:56.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.018+0000 7f1e647ed640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f1e4006be80 0x7f1e4006e340 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f1e58005880 tx=0x7f1e58009040 comp rx=0 tx=0).stop 2024-06-09T22:00:56.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0cf230 msgr2=0x7f1e5c1f0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:56.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0cf230 0x7f1e5c1f0690 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1e54031000 tx=0x7f1e54004a40 comp rx=0 tx=0).stop 2024-06-09T22:00:56.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 shutdown_connections 2024-06-09T22:00:56.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f1e5c0cf230 0x7f1e5c1f0690 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1e5c0ce7a0 0x7f1e5c1523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f1e4006be80 0x7f1e4006e340 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 --2- 172.21.15.32:0/2308318438 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f1e5c0ccbc0 0x7f1e5c151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 >> 172.21.15.32:0/2308318438 conn(0x7f1e5c0b1880 msgr2=0x7f1e5c154160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:56.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 shutdown_connections 2024-06-09T22:00:56.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.019+0000 7f1e647ed640 1 -- 172.21.15.32:0/2308318438 wait complete. 2024-06-09T22:00:56.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-06-09T22:00:56.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.138+0000 7efc804e7640 1 Processor -- start 2024-06-09T22:00:56.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.138+0000 7efc804e7640 1 -- start start 2024-06-09T22:00:56.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.139+0000 7efc804e7640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efc781546d0 0x7efc780c8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:56.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.139+0000 7efc804e7640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc7815e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:56.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.139+0000 7efc804e7640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc780c7ba0 con 0x7efc7806b390 2024-06-09T22:00:56.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.139+0000 7efc804e7640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7efc7815af10 con 0x7efc781546d0 2024-06-09T22:00:56.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.139+0000 7efc804e7640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7efc780c48d0 con 0x7efc7806ead0 2024-06-09T22:00:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.140+0000 7efc7e283640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efc781546d0 0x7efc780c8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.140+0000 7efc7ea84640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc7815e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.140+0000 7efc7e283640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7efc781546d0 0x7efc780c8640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:45538/0 (socket says 172.21.15.32:45538) 2024-06-09T22:00:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.140+0000 7efc7ea84640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc7815e1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.32:35924/0 (socket says 172.21.15.32:35924) 2024-06-09T22:00:56.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.140+0000 7efc7e283640 1 -- 172.21.15.32:0/2160444225 learned_addr learned my addr 172.21.15.32:0/2160444225 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:56.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.143+0000 7efc7e283640 1 -- 172.21.15.32:0/2160444225 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 msgr2=0x7efc7815e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:56.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.143+0000 7efc7e283640 1 --2- 172.21.15.32:0/2160444225 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc7815e1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.143+0000 7efc7e283640 1 -- 172.21.15.32:0/2160444225 >> v1:172.21.15.32:6789/0 conn(0x7efc7806b390 legacy=0x7efc7815dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:56.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.143+0000 7efc7e283640 1 -- 172.21.15.32:0/2160444225 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc7815b140 con 0x7efc781546d0 2024-06-09T22:00:56.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.145+0000 7efc7e283640 1 --2- 172.21.15.32:0/2160444225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 0x7efc780c8640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efc70009870 tx=0x7efc7002a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9aaac76619a3f93 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.148+0000 7efc7d281640 1 -- 172.21.15.32:0/2160444225 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70037070 con 0x7efc781546d0 2024-06-09T22:00:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.148+0000 7efc7d281640 1 -- 172.21.15.32:0/2160444225 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc70032440 con 0x7efc781546d0 2024-06-09T22:00:56.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.148+0000 7efc7d281640 1 -- 172.21.15.32:0/2160444225 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70032810 con 0x7efc781546d0 2024-06-09T22:00:56.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.149+0000 7efc804e7640 1 -- 172.21.15.32:0/2160444225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 msgr2=0x7efc780c8640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.149+0000 7efc804e7640 1 --2- 172.21.15.32:0/2160444225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 0x7efc780c8640 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7efc70009870 tx=0x7efc7002a230 comp rx=0 tx=0).stop 2024-06-09T22:00:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.149+0000 7efc804e7640 1 -- 172.21.15.32:0/2160444225 shutdown_connections 2024-06-09T22:00:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.149+0000 7efc804e7640 1 --2- 172.21.15.32:0/2160444225 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc7815e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.149+0000 7efc804e7640 1 --2- 172.21.15.32:0/2160444225 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 0x7efc780c8640 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.149+0000 7efc804e7640 1 -- 172.21.15.32:0/2160444225 >> 172.21.15.32:0/2160444225 conn(0x7efc780c9460 msgr2=0x7efc7806de30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.149+0000 7efc804e7640 1 -- 172.21.15.32:0/2160444225 shutdown_connections 2024-06-09T22:00:56.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.149+0000 7efc804e7640 1 -- 172.21.15.32:0/2160444225 wait complete. 2024-06-09T22:00:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.150+0000 7efc804e7640 1 Processor -- start 2024-06-09T22:00:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.150+0000 7efc804e7640 1 -- start start 2024-06-09T22:00:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.151+0000 7efc804e7640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7806b390 0x7efc7815cc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.151+0000 7efc804e7640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc78151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:56.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.151+0000 7efc804e7640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 0x7efc781f0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.151+0000 7efc804e7640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efc7815b420 con 0x7efc7806b390 2024-06-09T22:00:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.151+0000 7efc804e7640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7efc7815e720 con 0x7efc781546d0 2024-06-09T22:00:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.151+0000 7efc804e7640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7efc780c8b80 con 0x7efc7806ead0 2024-06-09T22:00:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.152+0000 7efc7da82640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc78151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.152+0000 7efc7ea84640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 0x7efc781f0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.152+0000 7efc7da82640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc78151460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3301/0 says I am v2:172.21.15.32:35936/0 (socket says 172.21.15.32:35936) 2024-06-09T22:00:56.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.152+0000 7efc7da82640 1 -- 172.21.15.32:0/2538093598 learned_addr learned my addr 172.21.15.32:0/2538093598 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:56.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.155+0000 7efc7da82640 1 -- 172.21.15.32:0/2538093598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 msgr2=0x7efc781f0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:56.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.155+0000 7efc7da82640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 0x7efc781f0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.155+0000 7efc7da82640 1 -- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7806b390 msgr2=0x7efc7815cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:56.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.155+0000 7efc7da82640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7806b390 0x7efc7815cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:56.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.155+0000 7efc7da82640 1 -- 172.21.15.32:0/2538093598 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc781f0670 con 0x7efc7806ead0 2024-06-09T22:00:56.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.158+0000 7efc7da82640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc78151460 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00c440 tx=0x7efc6c00c900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:56.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.162+0000 7efc76ffd640 1 -- 172.21.15.32:0/2538093598 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c017070 con 0x7efc7806ead0 2024-06-09T22:00:56.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.162+0000 7efc76ffd640 1 -- 172.21.15.32:0/2538093598 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc6c0035f0 con 0x7efc7806ead0 2024-06-09T22:00:56.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.162+0000 7efc76ffd640 1 -- 172.21.15.32:0/2538093598 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc6c0039e0 con 0x7efc7806ead0 2024-06-09T22:00:56.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.162+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc781e4ac0 con 0x7efc7806ead0 2024-06-09T22:00:56.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.162+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc781e4050 con 0x7efc7806ead0 2024-06-09T22:00:56.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.163+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc78068490 con 0x7efc7806ead0 2024-06-09T22:00:56.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.186+0000 7efc76ffd640 1 -- 172.21.15.32:0/2538093598 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7efc6c01a070 con 0x7efc7806ead0 2024-06-09T22:00:56.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.186+0000 7efc76ffd640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efc6406be80 0x7efc6406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:56.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.187+0000 7efc7e283640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efc6406be80 0x7efc6406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:56.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.187+0000 7efc7e283640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efc6406be80 0x7efc6406e340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc70034000 tx=0x7efc7002c000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:56.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.189+0000 7efc76ffd640 1 -- 172.21.15.32:0/2538093598 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7efc6c0569f0 con 0x7efc7806ead0 2024-06-09T22:00:56.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.444+0000 7efc76ffd640 1 -- 172.21.15.32:0/2538093598 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7efc6c021020 con 0x7efc7806ead0 2024-06-09T22:00:56.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:56.666+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7efc780c3940 con 0x7efc7806ead0 2024-06-09T22:00:57.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.340+0000 7efc76ffd640 1 -- 172.21.15.32:0/2538093598 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7efc6c0010c0 con 0x7efc7806ead0 2024-06-09T22:00:57.325 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 2 and data pool 3 2024-06-09T22:00:57.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efc6406be80 msgr2=0x7efc6406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efc6406be80 0x7efc6406e340 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7efc70034000 tx=0x7efc7002c000 comp rx=0 tx=0).stop 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 msgr2=0x7efc78151460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc78151460 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7efc6c00c440 tx=0x7efc6c00c900 comp rx=0 tx=0).stop 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 shutdown_connections 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7efc781546d0 0x7efc781f0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7efc6406be80 0x7efc6406e340 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7efc7806ead0 0x7efc78151460 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 --2- 172.21.15.32:0/2538093598 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc7806b390 0x7efc7815cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 >> 172.21.15.32:0/2538093598 conn(0x7efc780c9460 msgr2=0x7efc7806c2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.342+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 shutdown_connections 2024-06-09T22:00:57.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.343+0000 7efc804e7640 1 -- 172.21.15.32:0/2538093598 wait complete. 2024-06-09T22:00:57.338 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-06-09T22:00:57.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-06-09T22:00:57.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.465+0000 7fe5ffb28640 1 Processor -- start 2024-06-09T22:00:57.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.465+0000 7fe5ffb28640 1 -- start start 2024-06-09T22:00:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.466+0000 7fe5ffb28640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7fe5f8154b30 0x7fe5f815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.466+0000 7fe5ffb28640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe5f80cec00 0x7fe5f8164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:57.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.466+0000 7fe5ffb28640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5f80cd710 con 0x7fe5f80ce6b0 2024-06-09T22:00:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.466+0000 7fe5ffb28640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f80ca6e0 con 0x7fe5f8154b30 2024-06-09T22:00:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.466+0000 7fe5ffb28640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7fe5f815e030 con 0x7fe5f80cec00 2024-06-09T22:00:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.467+0000 7fe5fe0c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe5f80cec00 0x7fe5f8164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.468+0000 7fe5fe0c5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe5f80cec00 0x7fe5f8164f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:45552/0 (socket says 172.21.15.32:45552) 2024-06-09T22:00:57.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.468+0000 7fe5fe0c5640 1 -- 172.21.15.32:0/3099493057 learned_addr learned my addr 172.21.15.32:0/3099493057 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:57.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.470+0000 7fe5fe0c5640 1 -- 172.21.15.32:0/3099493057 >> v2:172.21.15.32:3300/0 conn(0x7fe5f8154b30 msgr2=0x7fe5f815e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:57.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.470+0000 7fe5fe0c5640 1 --2- 172.21.15.32:0/3099493057 >> v2:172.21.15.32:3300/0 conn(0x7fe5f8154b30 0x7fe5f815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.470+0000 7fe5fe0c5640 1 -- 172.21.15.32:0/3099493057 >> v1:172.21.15.32:6789/0 conn(0x7fe5f80ce6b0 legacy=0x7fe5f80cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:57.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.470+0000 7fe5fe0c5640 1 -- 172.21.15.32:0/3099493057 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f815dd00 con 0x7fe5f80cec00 2024-06-09T22:00:57.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.472+0000 7fe5fe0c5640 1 --2- 172.21.15.32:0/3099493057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f80cec00 0x7fe5f8164f60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80cd8e0 tx=0x7fe5f002a640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=377b260e0e12e641 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:57.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.475+0000 7fe5fc8c2640 1 -- 172.21.15.32:0/3099493057 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0036070 con 0x7fe5f80cec00 2024-06-09T22:00:57.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.475+0000 7fe5fc8c2640 1 -- 172.21.15.32:0/3099493057 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f0031600 con 0x7fe5f80cec00 2024-06-09T22:00:57.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.475+0000 7fe5fc8c2640 1 -- 172.21.15.32:0/3099493057 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f00319d0 con 0x7fe5f80cec00 2024-06-09T22:00:57.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.476+0000 7fe5ffb28640 1 -- 172.21.15.32:0/3099493057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f80cec00 msgr2=0x7fe5f8164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:57.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.476+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/3099493057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f80cec00 0x7fe5f8164f60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe5f80cd8e0 tx=0x7fe5f002a640 comp rx=0 tx=0).stop 2024-06-09T22:00:57.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.476+0000 7fe5ffb28640 1 -- 172.21.15.32:0/3099493057 shutdown_connections 2024-06-09T22:00:57.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.476+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/3099493057 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f80cec00 0x7fe5f8164f60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.476+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/3099493057 >> v2:172.21.15.32:3300/0 conn(0x7fe5f8154b30 0x7fe5f815e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.476+0000 7fe5ffb28640 1 -- 172.21.15.32:0/3099493057 >> 172.21.15.32:0/3099493057 conn(0x7fe5f80b1870 msgr2=0x7fe5f80b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:57.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.476+0000 7fe5ffb28640 1 -- 172.21.15.32:0/3099493057 shutdown_connections 2024-06-09T22:00:57.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.476+0000 7fe5ffb28640 1 -- 172.21.15.32:0/3099493057 wait complete. 2024-06-09T22:00:57.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.477+0000 7fe5ffb28640 1 Processor -- start 2024-06-09T22:00:57.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.477+0000 7fe5ffb28640 1 -- start start 2024-06-09T22:00:57.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.478+0000 7fe5ffb28640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f80ce6b0 0x7fe5f81e5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:57.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.478+0000 7fe5ffb28640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe5f80cec00 0x7fe5f81f01d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:57.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.478+0000 7fe5ffb28640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f8154b30 0x7fe5f81f4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:57.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.478+0000 7fe5ffb28640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f80caef0 con 0x7fe5f80ce6b0 2024-06-09T22:00:57.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.478+0000 7fe5ffb28640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe5f80cac10 con 0x7fe5f8154b30 2024-06-09T22:00:57.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.478+0000 7fe5ffb28640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fe5f815ea00 con 0x7fe5f80cec00 2024-06-09T22:00:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.479+0000 7fe5fd0c3640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe5f80cec00 0x7fe5f81f01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.479+0000 7fe5fe0c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f8154b30 0x7fe5f81f4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.479+0000 7fe5fe0c5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f8154b30 0x7fe5f81f4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:45564/0 (socket says 172.21.15.32:45564) 2024-06-09T22:00:57.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.479+0000 7fe5fe0c5640 1 -- 172.21.15.32:0/1426133034 learned_addr learned my addr 172.21.15.32:0/1426133034 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:57.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.482+0000 7fe5fe0c5640 1 -- 172.21.15.32:0/1426133034 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe5f80cec00 msgr2=0x7fe5f81f01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:57.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.482+0000 7fe5fe0c5640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe5f80cec00 0x7fe5f81f01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.483+0000 7fe5fe0c5640 1 -- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f80ce6b0 msgr2=0x7fe5f81e5910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:57.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.483+0000 7fe5fe0c5640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f80ce6b0 0x7fe5f81e5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:57.467 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.483+0000 7fe5fe0c5640 1 -- 172.21.15.32:0/1426133034 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5f80c4940 con 0x7fe5f8154b30 2024-06-09T22:00:57.468 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.485+0000 7fe5fe0c5640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f8154b30 0x7fe5f81f4190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00093e0 tx=0x7fe5f0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:57.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.488+0000 7fe5ea7fc640 1 -- 172.21.15.32:0/1426133034 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0041020 con 0x7fe5f8154b30 2024-06-09T22:00:57.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.489+0000 7fe5ea7fc640 1 -- 172.21.15.32:0/1426133034 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe5f0037040 con 0x7fe5f8154b30 2024-06-09T22:00:57.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.489+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5f80c4be0 con 0x7fe5f8154b30 2024-06-09T22:00:57.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.489+0000 7fe5ea7fc640 1 -- 172.21.15.32:0/1426133034 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe5f0036070 con 0x7fe5f8154b30 2024-06-09T22:00:57.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.489+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5f81f3c70 con 0x7fe5f8154b30 2024-06-09T22:00:57.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.490+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe5f81f3e70 con 0x7fe5f8154b30 2024-06-09T22:00:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.636+0000 7fe5ea7fc640 1 -- 172.21.15.32:0/1426133034 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe5f0044020 con 0x7fe5f8154b30 2024-06-09T22:00:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.636+0000 7fe5ea7fc640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe5dc06be80 0x7fe5dc06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.637+0000 7fe5fd8c4640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe5dc06be80 0x7fe5dc06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:57.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.637+0000 7fe5fd8c4640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe5dc06be80 0x7fe5dc06e340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009710 tx=0x7fe5f4006c90 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:57.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.759+0000 7fe5ea7fc640 1 -- 172.21.15.32:0/1426133034 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe5f0078300 con 0x7fe5f8154b30 2024-06-09T22:00:57.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.769+0000 7fe5ea7fc640 1 -- 172.21.15.32:0/1426133034 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7fe5f81f3e70 con 0x7fe5f8154b30 2024-06-09T22:00:57.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:57.975+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7fe5f8068490 con 0x7fe5f8154b30 2024-06-09T22:00:59.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.061+0000 7fe5ea7fc640 1 -- 172.21.15.32:0/1426133034 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v22) ==== 157+0+0 (secure 0 0 0) 0x7fe5f004b030 con 0x7fe5f8154b30 2024-06-09T22:00:59.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe5dc06be80 msgr2=0x7fe5dc06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe5dc06be80 0x7fe5dc06e340 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fe5f4009710 tx=0x7fe5f4006c90 comp rx=0 tx=0).stop 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f8154b30 msgr2=0x7fe5f81f4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f8154b30 0x7fe5f81f4190 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe5f00093e0 tx=0x7fe5f0004a40 comp rx=0 tx=0).stop 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 shutdown_connections 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe5f8154b30 0x7fe5f81f4190 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe5dc06be80 0x7fe5dc06e340 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe5f80cec00 0x7fe5f81f01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 --2- 172.21.15.32:0/1426133034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe5f80ce6b0 0x7fe5f81e5910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 >> 172.21.15.32:0/1426133034 conn(0x7fe5f80b1870 msgr2=0x7fe5f806c9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:59.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 shutdown_connections 2024-06-09T22:00:59.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.063+0000 7fe5ffb28640 1 -- 172.21.15.32:0/1426133034 wait complete. 2024-06-09T22:00:59.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-06-09T22:00:59.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.188+0000 7f967977e640 1 Processor -- start 2024-06-09T22:00:59.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.188+0000 7f967977e640 1 -- start start 2024-06-09T22:00:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.189+0000 7f967977e640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f96740caa10 0x7f9674070c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.189+0000 7f967977e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f967406fc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.189+0000 7f967977e640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f967415dd60 con 0x7f96740cadf0 2024-06-09T22:00:59.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.189+0000 7f967977e640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f967415e040 con 0x7f96740caa10 2024-06-09T22:00:59.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.189+0000 7f967977e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f967415e320 con 0x7f9674071150 2024-06-09T22:00:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.233+0000 7f96737fe640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f96740cadf0 0x7f96741541b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.32:47438/0 (socket says 172.21.15.32:47438) 2024-06-09T22:00:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.233+0000 7f96737fe640 1 -- 172.21.15.32:0/2865016378 learned_addr learned my addr 172.21.15.32:0/2865016378 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:00:59.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.233+0000 7f96727fc640 1 --2- 172.21.15.32:0/2865016378 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f967406fc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.237+0000 7f96727fc640 1 -- 172.21.15.32:0/2865016378 >> v2:172.21.15.32:3300/0 conn(0x7f96740caa10 msgr2=0x7f9674070c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:00:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.237+0000 7f96727fc640 1 --2- 172.21.15.32:0/2865016378 >> v2:172.21.15.32:3300/0 conn(0x7f96740caa10 0x7f9674070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.237+0000 7f96727fc640 1 -- 172.21.15.32:0/2865016378 >> v1:172.21.15.148:6789/0 conn(0x7f96740cadf0 legacy=0x7f96741541b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:59.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.237+0000 7f96727fc640 1 -- 172.21.15.32:0/2865016378 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f967415e600 con 0x7f9674071150 2024-06-09T22:00:59.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.239+0000 7f96727fc640 1 --2- 172.21.15.32:0/2865016378 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f967406fc20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f96680097a0 tx=0x7f966802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae5ca548cf5ee225 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.243+0000 7f9671ffb640 1 -- 172.21.15.32:0/2865016378 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668037070 con 0x7f9674071150 2024-06-09T22:00:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.243+0000 7f9671ffb640 1 -- 172.21.15.32:0/2865016378 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9668032440 con 0x7f9674071150 2024-06-09T22:00:59.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.243+0000 7f9671ffb640 1 -- 172.21.15.32:0/2865016378 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9668032810 con 0x7f9674071150 2024-06-09T22:00:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.244+0000 7f967977e640 1 -- 172.21.15.32:0/2865016378 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 msgr2=0x7f967406fc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.244+0000 7f967977e640 1 --2- 172.21.15.32:0/2865016378 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f967406fc20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f96680097a0 tx=0x7f966802a180 comp rx=0 tx=0).stop 2024-06-09T22:00:59.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.244+0000 7f967977e640 1 -- 172.21.15.32:0/2865016378 shutdown_connections 2024-06-09T22:00:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.244+0000 7f967977e640 1 --2- 172.21.15.32:0/2865016378 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f967406fc20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.244+0000 7f967977e640 1 --2- 172.21.15.32:0/2865016378 >> v2:172.21.15.32:3300/0 conn(0x7f96740caa10 0x7f9674070c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.244+0000 7f967977e640 1 -- 172.21.15.32:0/2865016378 >> 172.21.15.32:0/2865016378 conn(0x7f9674062660 msgr2=0x7f96740717c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:00:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.244+0000 7f967977e640 1 -- 172.21.15.32:0/2865016378 shutdown_connections 2024-06-09T22:00:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.244+0000 7f967977e640 1 -- 172.21.15.32:0/2865016378 wait complete. 2024-06-09T22:00:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.246+0000 7f967977e640 1 Processor -- start 2024-06-09T22:00:59.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.246+0000 7f967977e640 1 -- start start 2024-06-09T22:00:59.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.246+0000 7f967977e640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f96741f0e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:59.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.246+0000 7f967977e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96740caa10 0x7f9674158b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:59.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.246+0000 7f967977e640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96740cadf0 0x7f967414a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:59.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.246+0000 7f967977e640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f96740c88e0 con 0x7f96740cadf0 2024-06-09T22:00:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.246+0000 7f967977e640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f96740c8760 con 0x7f96740caa10 2024-06-09T22:00:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.246+0000 7f967977e640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f96740c8a60 con 0x7f9674071150 2024-06-09T22:00:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.247+0000 7f96737fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96740cadf0 0x7f967414a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.247+0000 7f96727fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96740caa10 0x7f9674158b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.247+0000 7f9672ffd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f96741f0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:59.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.247+0000 7f96737fe640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96740cadf0 0x7f967414a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55494/0 (socket says 172.21.15.32:55494) 2024-06-09T22:00:59.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.247+0000 7f96737fe640 1 -- 172.21.15.32:0/1748615611 learned_addr learned my addr 172.21.15.32:0/1748615611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:00:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.251+0000 7f9672ffd640 1 -- 172.21.15.32:0/1748615611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96740caa10 msgr2=0x7f9674158b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.251+0000 7f9672ffd640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96740caa10 0x7f9674158b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.251+0000 7f9672ffd640 1 -- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96740cadf0 msgr2=0x7f967414a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:00:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.251+0000 7f9672ffd640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96740cadf0 0x7f967414a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:00:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.251+0000 7f9672ffd640 1 -- 172.21.15.32:0/1748615611 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9674151760 con 0x7f9674071150 2024-06-09T22:00:59.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.251+0000 7f96727fc640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96740caa10 0x7f9674158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:00:59.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.253+0000 7f9672ffd640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f96741f0e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f966400b420 tx=0x7f966400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.257+0000 7f965b7fe640 1 -- 172.21.15.32:0/1748615611 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9664011830 con 0x7f9674071150 2024-06-09T22:00:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.257+0000 7f965b7fe640 1 -- 172.21.15.32:0/1748615611 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9664016070 con 0x7f9674071150 2024-06-09T22:00:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.257+0000 7f965b7fe640 1 -- 172.21.15.32:0/1748615611 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9664011c10 con 0x7f9674071150 2024-06-09T22:00:59.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.257+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9674151a40 con 0x7f9674071150 2024-06-09T22:00:59.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.258+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f96741f0650 con 0x7f9674071150 2024-06-09T22:00:59.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.258+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96740b3330 con 0x7f9674071150 2024-06-09T22:00:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.278+0000 7f965b7fe640 1 -- 172.21.15.32:0/1748615611 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f966401b460 con 0x7f9674071150 2024-06-09T22:00:59.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.279+0000 7f965b7fe640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f966006be80 0x7f966006e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:00:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.279+0000 7f96727fc640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f966006be80 0x7f966006e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:00:59.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.279+0000 7f96727fc640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f966006be80 0x7f966006e340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9668034000 tx=0x7f966802e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:00:59.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.281+0000 7f965b7fe640 1 -- 172.21.15.32:0/1748615611 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(22..22 src has 1..22) ==== 6007+0+0 (secure 0 0 0) 0x7f96640578b0 con 0x7f9674071150 2024-06-09T22:00:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.530+0000 7f965b7fe640 1 -- 172.21.15.32:0/1748615611 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f96740b3330 con 0x7f9674071150 2024-06-09T22:00:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:00:59.747+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f96740c9300 con 0x7f9674071150 2024-06-09T22:00:59.842 INFO:tasks.ceph.mds.c.smithi032.stderr:==00:00:00:16.689 41616== 2024-06-09T22:00:59.842 INFO:tasks.ceph.mds.c.smithi032.stderr:==00:00:00:16.689 41616== Exit program on first error (--exit-on-first-error=yes) 2024-06-09T22:00:59.966 INFO:tasks.ceph.mds.c.smithi032.stderr:daemon-helper: command failed with exit status 42 2024-06-09T22:01:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.021+0000 7f965b7fe640 1 -- 172.21.15.32:0/1748615611 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v24) ==== 228+0+0 (secure 0 0 0) 0x7f9664059810 con 0x7f9674071150 2024-06-09T22:01:02.005 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data_ec' created 2024-06-09T22:01:02.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.023+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f966006be80 msgr2=0x7f966006e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:02.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.023+0000 7f967977e640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f966006be80 0x7f966006e340 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f9668034000 tx=0x7f966802e000 comp rx=0 tx=0).stop 2024-06-09T22:01:02.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.024+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 msgr2=0x7f96741f0e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:02.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.024+0000 7f967977e640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f96741f0e50 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f966400b420 tx=0x7f966400b8e0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.025+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 shutdown_connections 2024-06-09T22:01:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.025+0000 7f967977e640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96740cadf0 0x7f967414a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.025+0000 7f967977e640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f96740caa10 0x7f9674158b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.025+0000 7f967977e640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f966006be80 0x7f966006e340 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.025+0000 7f967977e640 1 --2- 172.21.15.32:0/1748615611 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f9674071150 0x7f96741f0e50 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.025+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 >> 172.21.15.32:0/1748615611 conn(0x7f9674062660 msgr2=0x7f9674074820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.025+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 shutdown_connections 2024-06-09T22:01:02.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.026+0000 7f967977e640 1 -- 172.21.15.32:0/1748615611 wait complete. 2024-06-09T22:01:02.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-06-09T22:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.146+0000 7ff776cf5640 1 Processor -- start 2024-06-09T22:01:02.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.146+0000 7ff776cf5640 1 -- start start 2024-06-09T22:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.147+0000 7ff776cf5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff77015a530 0x7ff7700c8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.147+0000 7ff776cf5640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff7700c90a0 0x7ff7700ca550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.147+0000 7ff776cf5640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff77015aa70 con 0x7ff7701546e0 2024-06-09T22:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.147+0000 7ff776cf5640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7ff7700c94f0 con 0x7ff7700c90a0 2024-06-09T22:01:02.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.147+0000 7ff776cf5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7ff77015ad10 con 0x7ff77015a530 2024-06-09T22:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.148+0000 7ff775292640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7ff7700c90a0 0x7ff7700ca550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.148+0000 7ff767fff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff77015a530 0x7ff7700c8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.148+0000 7ff774a91640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7ff7701546e0 0x7ff7700cdcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.32:50216/0 (socket says 172.21.15.32:50216) 2024-06-09T22:01:02.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.148+0000 7ff774a91640 1 -- 172.21.15.32:0/1231821692 learned_addr learned my addr 172.21.15.32:0/1231821692 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:01:02.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.150+0000 7ff775292640 1 -- 172.21.15.32:0/1231821692 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff77015a530 msgr2=0x7ff7700c8b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:02.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.150+0000 7ff775292640 1 --2- 172.21.15.32:0/1231821692 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff77015a530 0x7ff7700c8b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.150+0000 7ff775292640 1 -- 172.21.15.32:0/1231821692 >> v1:172.21.15.148:6789/0 conn(0x7ff7701546e0 legacy=0x7ff7700cdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:02.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.150+0000 7ff775292640 1 -- 172.21.15.32:0/1231821692 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff77015aff0 con 0x7ff7700c90a0 2024-06-09T22:01:02.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.150+0000 7ff767fff640 1 --2- 172.21.15.32:0/1231821692 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff77015a530 0x7ff7700c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:01:02.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.152+0000 7ff775292640 1 --2- 172.21.15.32:0/1231821692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff7700ca550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff768009d90 tx=0x7ff76802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf828de3ca8cce8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.155+0000 7ff7677fe640 1 -- 172.21.15.32:0/1231821692 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768007740 con 0x7ff7700c90a0 2024-06-09T22:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff7677fe640 1 -- 172.21.15.32:0/1231821692 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7680078e0 con 0x7ff7700c90a0 2024-06-09T22:01:02.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff7677fe640 1 -- 172.21.15.32:0/1231821692 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff768007cb0 con 0x7ff7700c90a0 2024-06-09T22:01:02.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff776cf5640 1 -- 172.21.15.32:0/1231821692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 msgr2=0x7ff7700ca550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:02.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1231821692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff7700ca550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff768009d90 tx=0x7ff76802a660 comp rx=0 tx=0).stop 2024-06-09T22:01:02.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff776cf5640 1 -- 172.21.15.32:0/1231821692 shutdown_connections 2024-06-09T22:01:02.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1231821692 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff7700ca550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1231821692 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff77015a530 0x7ff7700c8b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff776cf5640 1 -- 172.21.15.32:0/1231821692 >> 172.21.15.32:0/1231821692 conn(0x7ff77006f800 msgr2=0x7ff77006fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:02.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff776cf5640 1 -- 172.21.15.32:0/1231821692 shutdown_connections 2024-06-09T22:01:02.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.156+0000 7ff776cf5640 1 -- 172.21.15.32:0/1231821692 wait complete. 2024-06-09T22:01:02.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.158+0000 7ff776cf5640 1 Processor -- start 2024-06-09T22:01:02.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.158+0000 7ff776cf5640 1 -- start start 2024-06-09T22:01:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.158+0000 7ff776cf5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff77014cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.158+0000 7ff776cf5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff7701546e0 0x7ff7701f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.158+0000 7ff776cf5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff77015a530 0x7ff7701f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.159+0000 7ff776cf5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff770154ac0 con 0x7ff7700c90a0 2024-06-09T22:01:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.159+0000 7ff776cf5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7ff7700ca030 con 0x7ff77015a530 2024-06-09T22:01:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.159+0000 7ff776cf5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7ff770154c40 con 0x7ff7701546e0 2024-06-09T22:01:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.159+0000 7ff774a91640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff77014cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:02.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.159+0000 7ff767fff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff7701546e0 0x7ff7701f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.159+0000 7ff774a91640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff77014cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59756/0 (socket says 172.21.15.32:59756) 2024-06-09T22:01:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.159+0000 7ff774a91640 1 -- 172.21.15.32:0/1963740143 learned_addr learned my addr 172.21.15.32:0/1963740143 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:01:02.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.161+0000 7ff775292640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff77015a530 0x7ff7701f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.163+0000 7ff774a91640 1 -- 172.21.15.32:0/1963740143 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff7701546e0 msgr2=0x7ff7701f0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.163+0000 7ff774a91640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff7701546e0 0x7ff7701f0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.163+0000 7ff774a91640 1 -- 172.21.15.32:0/1963740143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff77015a530 msgr2=0x7ff7701f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.163+0000 7ff774a91640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff77015a530 0x7ff7701f34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.163+0000 7ff774a91640 1 -- 172.21.15.32:0/1963740143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7701f3a10 con 0x7ff7700c90a0 2024-06-09T22:01:02.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.165+0000 7ff774a91640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff77014cbd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00c400 tx=0x7ff76c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.170+0000 7ff7657fa640 1 -- 172.21.15.32:0/1963740143 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c017070 con 0x7ff7700c90a0 2024-06-09T22:01:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.170+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7701eff60 con 0x7ff7700c90a0 2024-06-09T22:01:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.171+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7701f3f80 con 0x7ff7700c90a0 2024-06-09T22:01:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.171+0000 7ff7657fa640 1 -- 172.21.15.32:0/1963740143 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff76c005ce0 con 0x7ff7700c90a0 2024-06-09T22:01:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.171+0000 7ff7657fa640 1 -- 172.21.15.32:0/1963740143 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff76c021060 con 0x7ff7700c90a0 2024-06-09T22:01:02.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.171+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff750005290 con 0x7ff7700c90a0 2024-06-09T22:01:02.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.457+0000 7ff7657fa640 1 -- 172.21.15.32:0/1963740143 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7ff76c01a070 con 0x7ff7700c90a0 2024-06-09T22:01:02.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.457+0000 7ff7657fa640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7ff75c06be30 0x7ff75c06e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:02.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.458+0000 7ff767fff640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7ff75c06be30 0x7ff75c06e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:02.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.458+0000 7ff767fff640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7ff75c06be30 0x7ff75c06e2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff760009560 tx=0x7ff760009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.473+0000 7ff7657fa640 1 -- 172.21.15.32:0/1963740143 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6500+0+0 (secure 0 0 0) 0x7ff76c056d20 con 0x7ff7700c90a0 2024-06-09T22:01:02.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.473+0000 7ff7657fa640 1 -- 172.21.15.32:0/1963740143 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7ff76c058c80 con 0x7ff7700c90a0 2024-06-09T22:01:02.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.659+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7ff750005580 con 0x7ff7700c90a0 2024-06-09T22:01:02.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.992+0000 7ff7657fa640 1 -- 172.21.15.32:0/1963740143 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v25) ==== 169+0+0 (secure 0 0 0) 0x7ff76c05cf80 con 0x7ff7700c90a0 2024-06-09T22:01:02.977 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 4 allow_ec_overwrites to true 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.994+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7ff75c06be30 msgr2=0x7ff75c06e2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.994+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7ff75c06be30 0x7ff75c06e2f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff760009560 tx=0x7ff760009190 comp rx=0 tx=0).stop 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.994+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 msgr2=0x7ff77014cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.994+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff77014cbd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff76c00c400 tx=0x7ff76c00c8c0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.995+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 shutdown_connections 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.995+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7ff77015a530 0x7ff7701f34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.995+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7ff75c06be30 0x7ff75c06e2f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.995+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7ff7701546e0 0x7ff7701f0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.995+0000 7ff776cf5640 1 --2- 172.21.15.32:0/1963740143 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff7700c90a0 0x7ff77014cbd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:02.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.995+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 >> 172.21.15.32:0/1963740143 conn(0x7ff77006f800 msgr2=0x7ff7700cdcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:02.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.995+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 shutdown_connections 2024-06-09T22:01:02.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:02.995+0000 7ff776cf5640 1 -- 172.21.15.32:0/1963740143 wait complete. 2024-06-09T22:01:02.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-06-09T22:01:03.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.125+0000 7f2bef283640 1 Processor -- start 2024-06-09T22:01:03.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.126+0000 7f2bef283640 1 -- start start 2024-06-09T22:01:03.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.126+0000 7f2bef283640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2be80ccbc0 0x7f2be80ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.126+0000 7f2bef283640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80ce090 0x7f2be80ce540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.126+0000 7f2bef283640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be815a9f0 con 0x7f2be80cac30 2024-06-09T22:01:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.126+0000 7f2bef283640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f2be80cea80 con 0x7f2be80ccbc0 2024-06-09T22:01:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.126+0000 7f2bef283640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f2be815acb0 con 0x7f2be80ce090 2024-06-09T22:01:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.127+0000 7f2bed01f640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2be80ccbc0 0x7f2be80ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.127+0000 7f2bec81e640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2be80cac30 0x7f2be80cf230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33164/0 (socket says 172.21.15.32:33164) 2024-06-09T22:01:03.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.127+0000 7f2bec81e640 1 -- 172.21.15.32:0/2788145647 learned_addr learned my addr 172.21.15.32:0/2788145647 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:01:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.133+0000 7f2bed01f640 1 -- 172.21.15.32:0/2788145647 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80ce090 msgr2=0x7f2be80ce540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.133+0000 7f2bed01f640 1 --2- 172.21.15.32:0/2788145647 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80ce090 0x7f2be80ce540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.133+0000 7f2bed01f640 1 -- 172.21.15.32:0/2788145647 >> v1:172.21.15.32:6789/0 conn(0x7f2be80cac30 legacy=0x7f2be80cf230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:03.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.133+0000 7f2bed01f640 1 -- 172.21.15.32:0/2788145647 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be815af90 con 0x7f2be80ccbc0 2024-06-09T22:01:03.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.138+0000 7f2bed01f640 1 --2- 172.21.15.32:0/2788145647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2be80ccbc0 0x7f2be80ca6f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2be40097a0 tx=0x7f2be402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ca87b079fe59cd server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:03.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.145+0000 7f2bdffff640 1 -- 172.21.15.32:0/2788145647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4037070 con 0x7f2be80ccbc0 2024-06-09T22:01:03.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.145+0000 7f2bdffff640 1 -- 172.21.15.32:0/2788145647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2be4032440 con 0x7f2be80ccbc0 2024-06-09T22:01:03.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.145+0000 7f2bdffff640 1 -- 172.21.15.32:0/2788145647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2be4032850 con 0x7f2be80ccbc0 2024-06-09T22:01:03.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.146+0000 7f2bef283640 1 -- 172.21.15.32:0/2788145647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2be80ccbc0 msgr2=0x7f2be80ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:03.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.146+0000 7f2bef283640 1 --2- 172.21.15.32:0/2788145647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2be80ccbc0 0x7f2be80ca6f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f2be40097a0 tx=0x7f2be402a250 comp rx=0 tx=0).stop 2024-06-09T22:01:03.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.146+0000 7f2bef283640 1 -- 172.21.15.32:0/2788145647 shutdown_connections 2024-06-09T22:01:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.146+0000 7f2bef283640 1 --2- 172.21.15.32:0/2788145647 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80ce090 0x7f2be80ce540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.146+0000 7f2bef283640 1 --2- 172.21.15.32:0/2788145647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2be80ccbc0 0x7f2be80ca6f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.146+0000 7f2bef283640 1 -- 172.21.15.32:0/2788145647 >> 172.21.15.32:0/2788145647 conn(0x7f2be80b1880 msgr2=0x7f2be80b1c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.146+0000 7f2bef283640 1 -- 172.21.15.32:0/2788145647 shutdown_connections 2024-06-09T22:01:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.146+0000 7f2bef283640 1 -- 172.21.15.32:0/2788145647 wait complete. 2024-06-09T22:01:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.147+0000 7f2bef283640 1 Processor -- start 2024-06-09T22:01:03.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.148+0000 7f2bef283640 1 -- start start 2024-06-09T22:01:03.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.148+0000 7f2bef283640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80cac30 0x7f2be81f4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:03.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.148+0000 7f2bef283640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2be80ccbc0 0x7f2be8155050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:03.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.148+0000 7f2bef283640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2be80ce090 0x7f2be81f15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:03.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.148+0000 7f2bef283640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80cdb70 con 0x7f2be80ce090 2024-06-09T22:01:03.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.148+0000 7f2bef283640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2be815b550 con 0x7f2be80ccbc0 2024-06-09T22:01:03.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.148+0000 7f2bef283640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f2be815a6e0 con 0x7f2be80cac30 2024-06-09T22:01:03.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.152+0000 7f2bec81e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2be80ccbc0 0x7f2be8155050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:03.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.153+0000 7f2bec81e640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2be80ccbc0 0x7f2be8155050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:39686/0 (socket says 172.21.15.32:39686) 2024-06-09T22:01:03.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.153+0000 7f2bec81e640 1 -- 172.21.15.32:0/3567611180 learned_addr learned my addr 172.21.15.32:0/3567611180 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:01:03.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.159+0000 7f2bed01f640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80cac30 0x7f2be81f4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.160+0000 7f2bec81e640 1 -- 172.21.15.32:0/3567611180 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80cac30 msgr2=0x7f2be81f4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.160+0000 7f2bec81e640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80cac30 0x7f2be81f4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.160+0000 7f2bec81e640 1 -- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2be80ce090 msgr2=0x7f2be81f15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.160+0000 7f2bec81e640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2be80ce090 0x7f2be81f15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:03.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.160+0000 7f2bec81e640 1 -- 172.21.15.32:0/3567611180 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be81f0b00 con 0x7f2be80ccbc0 2024-06-09T22:01:03.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.167+0000 7f2bec81e640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2be80ccbc0 0x7f2be8155050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800c440 tx=0x7f2bd800c900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.171+0000 7f2bddffb640 1 -- 172.21.15.32:0/3567611180 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8017070 con 0x7f2be80ccbc0 2024-06-09T22:01:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.171+0000 7f2bddffb640 1 -- 172.21.15.32:0/3567611180 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd8005ce0 con 0x7f2be80ccbc0 2024-06-09T22:01:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.171+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be8157c10 con 0x7f2be80ccbc0 2024-06-09T22:01:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.171+0000 7f2bddffb640 1 -- 172.21.15.32:0/3567611180 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd80072f0 con 0x7f2be80ccbc0 2024-06-09T22:01:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.172+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be81f0dc0 con 0x7f2be80ccbc0 2024-06-09T22:01:03.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.173+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be8068490 con 0x7f2be80ccbc0 2024-06-09T22:01:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.195+0000 7f2bddffb640 1 -- 172.21.15.32:0/3567611180 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2bd801a070 con 0x7f2be80ccbc0 2024-06-09T22:01:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.196+0000 7f2bddffb640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2bcc06be80 0x7f2bcc06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.196+0000 7f2bed01f640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2bcc06be80 0x7f2bcc06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:03.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.196+0000 7f2bed01f640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2bcc06be80 0x7f2bcc06e340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2be4034000 tx=0x7f2be402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:03.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.198+0000 7f2bddffb640 1 -- 172.21.15.32:0/3567611180 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 6500+0+0 (secure 0 0 0) 0x7f2bd8056960 con 0x7f2be80ccbc0 2024-06-09T22:01:03.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.471+0000 7f2bddffb640 1 -- 172.21.15.32:0/3567611180 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f2bd8021020 con 0x7f2be80ccbc0 2024-06-09T22:01:03.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:03.695+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f2be806e270 con 0x7f2be80ccbc0 2024-06-09T22:01:04.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.162+0000 7f2bddffb640 1 -- 172.21.15.32:0/3567611180 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v7) ==== 137+0+0 (secure 0 0 0) 0x7f2bd8058fc0 con 0x7f2be80ccbc0 2024-06-09T22:01:04.147 INFO:teuthology.orchestra.run.smithi032.stderr:added data pool 4 to fsmap 2024-06-09T22:01:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2bcc06be80 msgr2=0x7f2bcc06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2bcc06be80 0x7f2bcc06e340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f2be4034000 tx=0x7f2be402e000 comp rx=0 tx=0).stop 2024-06-09T22:01:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2be80ccbc0 msgr2=0x7f2be8155050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2be80ccbc0 0x7f2be8155050 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800c440 tx=0x7f2bd800c900 comp rx=0 tx=0).stop 2024-06-09T22:01:04.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 shutdown_connections 2024-06-09T22:01:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2be80ce090 0x7f2be81f15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2be80ccbc0 0x7f2be8155050 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2bcc06be80 0x7f2bcc06e340 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 --2- 172.21.15.32:0/3567611180 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2be80cac30 0x7f2be81f4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 >> 172.21.15.32:0/3567611180 conn(0x7f2be80b1880 msgr2=0x7f2be81540a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 shutdown_connections 2024-06-09T22:01:04.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.165+0000 7f2bef283640 1 -- 172.21.15.32:0/3567611180 wait complete. 2024-06-09T22:01:04.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-09T22:01:04.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.279+0000 7f3d464fc640 1 Processor -- start 2024-06-09T22:01:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.279+0000 7f3d464fc640 1 -- start start 2024-06-09T22:01:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.280+0000 7f3d464fc640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f3d40151220 0x7f3d4015a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.280+0000 7f3d464fc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d400c8370 con 0x7f3d40157d70 2024-06-09T22:01:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.280+0000 7f3d464fc640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d40071ec0 con 0x7f3d40071a50 2024-06-09T22:01:04.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.280+0000 7f3d464fc640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7f3d401581e0 con 0x7f3d40151220 2024-06-09T22:01:04.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.281+0000 7f3d3f7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f3d40157d70 0x7f3d400c7c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33166/0 (socket says 172.21.15.32:33166) 2024-06-09T22:01:04.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.281+0000 7f3d3f7fe640 1 -- 172.21.15.32:0/1723195827 learned_addr learned my addr 172.21.15.32:0/1723195827 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:01:04.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.281+0000 7f3d3ffff640 1 --2- 172.21.15.32:0/1723195827 >> v2:172.21.15.148:3300/0 conn(0x7f3d40151220 0x7f3d4015a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.284+0000 7f3d3effd640 1 -- 172.21.15.32:0/1723195827 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 891175616 0 0) 0x7f3d400c8370 con 0x7f3d40157d70 2024-06-09T22:01:04.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.284+0000 7f3d3effd640 1 -- 172.21.15.32:0/1723195827 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d28005730 con 0x7f3d40157d70 2024-06-09T22:01:04.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.284+0000 7f3d3ffff640 1 -- 172.21.15.32:0/1723195827 >> v1:172.21.15.148:6789/0 conn(0x7f3d40071a50 legacy=0x7f3d4006b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.284+0000 7f3d3ffff640 1 -- 172.21.15.32:0/1723195827 >> v1:172.21.15.32:6789/0 conn(0x7f3d40157d70 legacy=0x7f3d400c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.284+0000 7f3d3ffff640 1 -- 172.21.15.32:0/1723195827 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d40072190 con 0x7f3d40151220 2024-06-09T22:01:04.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.287+0000 7f3d3ffff640 1 --2- 172.21.15.32:0/1723195827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 0x7f3d4015a810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d300097a0 tx=0x7f3d3002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=718c3b6694f05bd8 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:04.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d3effd640 1 -- 172.21.15.32:0/1723195827 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30037070 con 0x7f3d40151220 2024-06-09T22:01:04.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d3effd640 1 -- 172.21.15.32:0/1723195827 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d30032640 con 0x7f3d40151220 2024-06-09T22:01:04.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d3effd640 1 -- 172.21.15.32:0/1723195827 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d30032a10 con 0x7f3d40151220 2024-06-09T22:01:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d464fc640 1 -- 172.21.15.32:0/1723195827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 msgr2=0x7f3d4015a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d464fc640 1 --2- 172.21.15.32:0/1723195827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 0x7f3d4015a810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d300097a0 tx=0x7f3d3002a2c0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d464fc640 1 -- 172.21.15.32:0/1723195827 shutdown_connections 2024-06-09T22:01:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d464fc640 1 --2- 172.21.15.32:0/1723195827 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 0x7f3d4015a810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d464fc640 1 -- 172.21.15.32:0/1723195827 >> 172.21.15.32:0/1723195827 conn(0x7f3d400cbb70 msgr2=0x7f3d400cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d464fc640 1 -- 172.21.15.32:0/1723195827 shutdown_connections 2024-06-09T22:01:04.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.291+0000 7f3d464fc640 1 -- 172.21.15.32:0/1723195827 wait complete. 2024-06-09T22:01:04.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.292+0000 7f3d464fc640 1 Processor -- start 2024-06-09T22:01:04.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.293+0000 7f3d464fc640 1 -- start start 2024-06-09T22:01:04.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.293+0000 7f3d464fc640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3d40071a50 0x7f3d400caff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.293+0000 7f3d464fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 0x7f3d401e2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.293+0000 7f3d464fc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d40157d70 0x7f3d400c1230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.293+0000 7f3d464fc640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3d40157570 con 0x7f3d40157d70 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.293+0000 7f3d464fc640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f3d40072750 con 0x7f3d40151220 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.293+0000 7f3d464fc640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f3d40157850 con 0x7f3d40071a50 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.294+0000 7f3d44a99640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d40157d70 0x7f3d400c1230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.294+0000 7f3d3ffff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3d40071a50 0x7f3d400caff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.294+0000 7f3d3f7fe640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 0x7f3d401e2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.294+0000 7f3d44a99640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d40157d70 0x7f3d400c1230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59784/0 (socket says 172.21.15.32:59784) 2024-06-09T22:01:04.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.294+0000 7f3d44a99640 1 -- 172.21.15.32:0/2307180972 learned_addr learned my addr 172.21.15.32:0/2307180972 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:01:04.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.298+0000 7f3d3f7fe640 1 -- 172.21.15.32:0/2307180972 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3d40071a50 msgr2=0x7f3d400caff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.298+0000 7f3d3f7fe640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3d40071a50 0x7f3d400caff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.298+0000 7f3d3f7fe640 1 -- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d40157d70 msgr2=0x7f3d400c1230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.298+0000 7f3d3f7fe640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d40157d70 0x7f3d400c1230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.298+0000 7f3d3f7fe640 1 -- 172.21.15.32:0/2307180972 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d4014ea10 con 0x7f3d40151220 2024-06-09T22:01:04.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.300+0000 7f3d3f7fe640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 0x7f3d401e2d50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400c3c0 tx=0x7f3d34001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:04.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.303+0000 7f3d3cff9640 1 -- 172.21.15.32:0/2307180972 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d34017070 con 0x7f3d40151220 2024-06-09T22:01:04.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.303+0000 7f3d3cff9640 1 -- 172.21.15.32:0/2307180972 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d34005ce0 con 0x7f3d40151220 2024-06-09T22:01:04.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.303+0000 7f3d3cff9640 1 -- 172.21.15.32:0/2307180972 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d340072d0 con 0x7f3d40151220 2024-06-09T22:01:04.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.303+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d400cb810 con 0x7f3d40151220 2024-06-09T22:01:04.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.303+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d400c9c50 con 0x7f3d40151220 2024-06-09T22:01:04.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.304+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d40068490 con 0x7f3d40151220 2024-06-09T22:01:04.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.324+0000 7f3d3cff9640 1 -- 172.21.15.32:0/2307180972 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f3d3401a070 con 0x7f3d40151220 2024-06-09T22:01:04.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.324+0000 7f3d3cff9640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f3d2c06be80 0x7f3d2c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.324+0000 7f3d3ffff640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f3d2c06be80 0x7f3d2c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.325+0000 7f3d3ffff640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f3d2c06be80 0x7f3d2c06e340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d30034000 tx=0x7f3d3002e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:04.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.327+0000 7f3d3cff9640 1 -- 172.21.15.32:0/2307180972 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f3d34056a20 con 0x7f3d40151220 2024-06-09T22:01:04.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.581+0000 7f3d3cff9640 1 -- 172.21.15.32:0/2307180972 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f3d4014ea10 con 0x7f3d40151220 2024-06-09T22:01:04.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.802+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d40155370 con 0x7f3d40151220 2024-06-09T22:01:04.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.830+0000 7f3d3cff9640 1 -- 172.21.15.32:0/2307180972 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6834 (secure 0 0 0) 0x7f3d34058980 con 0x7f3d40151220 2024-06-09T22:01:04.814 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:01:04.814 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4199,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2870873470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2870873470},{"type":"v1","addr":"172.21.15.32:6835","nonce":2870873470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/1513914407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":1513914407},{"type":"v1","addr":"172.21.15.148:6833","nonce":1513914407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4243,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2699555510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":2699555510},{"type":"v1","addr":"172.21.15.148:6835","nonce":2699555510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2047518031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2047518031},{"type":"v1","addr":"172.21.15.148:6837","nonce":2047518031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2947563635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2947563635},{"type":"v1","addr":"172.21.15.32:6837","nonce":2947563635}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-09T22:00:56.720976+0000","modified":"2024-06-09T22:01:03.721677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/3734386334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":3734386334},{"type":"v1","addr":"172.21.15.32:6839","nonce":3734386334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4263,"qdb_cluster":[4263]},"id":1}]} 2024-06-09T22:01:04.815 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 7 2024-06-09T22:01:04.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f3d2c06be80 msgr2=0x7f3d2c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f3d2c06be80 0x7f3d2c06e340 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d30034000 tx=0x7f3d3002e000 comp rx=0 tx=0).stop 2024-06-09T22:01:04.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 msgr2=0x7f3d401e2d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 0x7f3d401e2d50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3d3400c3c0 tx=0x7f3d34001e70 comp rx=0 tx=0).stop 2024-06-09T22:01:04.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 shutdown_connections 2024-06-09T22:01:04.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3d40157d70 0x7f3d400c1230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f3d40151220 0x7f3d401e2d50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f3d2c06be80 0x7f3d2c06e340 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 --2- 172.21.15.32:0/2307180972 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f3d40071a50 0x7f3d400caff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 >> 172.21.15.32:0/2307180972 conn(0x7f3d400cbb70 msgr2=0x7f3d400cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:04.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 shutdown_connections 2024-06-09T22:01:04.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.832+0000 7f3d464fc640 1 -- 172.21.15.32:0/2307180972 wait complete. 2024-06-09T22:01:04.829 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-09T22:01:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.954+0000 7f779f49a640 1 Processor -- start 2024-06-09T22:01:04.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.954+0000 7f779f49a640 1 -- start start 2024-06-09T22:01:04.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.955+0000 7f779f49a640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.955+0000 7f779f49a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77980ce870 con 0x7f77980ce3e0 2024-06-09T22:01:04.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.955+0000 7f779f49a640 1 -- --> v1:172.21.15.148:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f779815a7d0 con 0x7f77980c95e0 2024-06-09T22:01:04.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.955+0000 7f779f49a640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f779815b6a0 con 0x7f77980cedc0 2024-06-09T22:01:04.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.955+0000 7f779d236640 1 --1- >> v1:172.21.15.148:6789/0 conn(0x7f77980c95e0 0x7f77980ca740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.148:6789/0 says I am v1:172.21.15.32:50242/0 (socket says 172.21.15.32:50242) 2024-06-09T22:01:04.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.955+0000 7f779d236640 1 -- 172.21.15.32:0/3448268922 learned_addr learned my addr 172.21.15.32:0/3448268922 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:01:04.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.957+0000 7f779ca35640 1 --2- 172.21.15.32:0/3448268922 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.958+0000 7f778ffff640 1 -- 172.21.15.32:0/3448268922 <== mon.1 v1:172.21.15.148:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4028548344 0 0) 0x7f779815a7d0 con 0x7f77980c95e0 2024-06-09T22:01:04.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.958+0000 7f778ffff640 1 -- 172.21.15.32:0/3448268922 --> v1:172.21.15.148:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7784003520 con 0x7f77980c95e0 2024-06-09T22:01:04.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.959+0000 7f779ca35640 1 -- 172.21.15.32:0/3448268922 >> v1:172.21.15.148:6789/0 conn(0x7f77980c95e0 legacy=0x7f77980ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.959+0000 7f779ca35640 1 -- 172.21.15.32:0/3448268922 >> v1:172.21.15.32:6789/0 conn(0x7f77980ce3e0 legacy=0x7f77980cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:04.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.960+0000 7f779ca35640 1 -- 172.21.15.32:0/3448268922 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f779815aa90 con 0x7f77980cedc0 2024-06-09T22:01:04.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.965+0000 7f779ca35640 1 --2- 172.21.15.32:0/3448268922 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980cdea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f77880097d0 tx=0x7f778802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9476c1d7afc1a73 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:04.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.970+0000 7f778ffff640 1 -- 172.21.15.32:0/3448268922 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788037070 con 0x7f77980cedc0 2024-06-09T22:01:04.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.970+0000 7f778ffff640 1 -- 172.21.15.32:0/3448268922 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7788032440 con 0x7f77980cedc0 2024-06-09T22:01:04.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.970+0000 7f778ffff640 1 -- 172.21.15.32:0/3448268922 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788032810 con 0x7f77980cedc0 2024-06-09T22:01:04.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.971+0000 7f779f49a640 1 -- 172.21.15.32:0/3448268922 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 msgr2=0x7f77980cdea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.971+0000 7f779f49a640 1 --2- 172.21.15.32:0/3448268922 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980cdea0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f77880097d0 tx=0x7f778802a250 comp rx=0 tx=0).stop 2024-06-09T22:01:04.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.971+0000 7f779f49a640 1 -- 172.21.15.32:0/3448268922 shutdown_connections 2024-06-09T22:01:04.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.971+0000 7f779f49a640 1 --2- 172.21.15.32:0/3448268922 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980cdea0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.971+0000 7f779f49a640 1 -- 172.21.15.32:0/3448268922 >> 172.21.15.32:0/3448268922 conn(0x7f77980b18d0 msgr2=0x7f77980b1ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:04.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.971+0000 7f779f49a640 1 -- 172.21.15.32:0/3448268922 shutdown_connections 2024-06-09T22:01:04.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.971+0000 7f779f49a640 1 -- 172.21.15.32:0/3448268922 wait complete. 2024-06-09T22:01:04.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.972+0000 7f779f49a640 1 Processor -- start 2024-06-09T22:01:04.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.972+0000 7f779f49a640 1 -- start start 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779f49a640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77980c95e0 0x7f77981f0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779f49a640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77980ce3e0 0x7f7798153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779f49a640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980c8870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779f49a640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f779815b330 con 0x7f77980ce3e0 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779f49a640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f779815b050 con 0x7f77980c95e0 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779f49a640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f77980c99c0 con 0x7f77980cedc0 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779da37640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980c8870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779ca35640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77980ce3e0 0x7f7798153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.973+0000 7f779d236640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77980c95e0 0x7f77981f0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:04.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.974+0000 7f779ca35640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77980ce3e0 0x7f7798153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59794/0 (socket says 172.21.15.32:59794) 2024-06-09T22:01:04.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.974+0000 7f779ca35640 1 -- 172.21.15.32:0/3818413460 learned_addr learned my addr 172.21.15.32:0/3818413460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:01:04.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.977+0000 7f779ca35640 1 -- 172.21.15.32:0/3818413460 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 msgr2=0x7f77980c8870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.977+0000 7f779ca35640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980c8870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.977+0000 7f779ca35640 1 -- 172.21.15.32:0/3818413460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77980c95e0 msgr2=0x7f77981f0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:04.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.977+0000 7f779ca35640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77980c95e0 0x7f77981f0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:04.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.977+0000 7f779ca35640 1 -- 172.21.15.32:0/3818413460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7798154280 con 0x7f77980ce3e0 2024-06-09T22:01:04.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.979+0000 7f779ca35640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77980ce3e0 0x7f7798153a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f77880098f0 tx=0x7f77880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:04.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.983+0000 7f778dffb640 1 -- 172.21.15.32:0/3818413460 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7788037050 con 0x7f77980ce3e0 2024-06-09T22:01:04.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.983+0000 7f778dffb640 1 -- 172.21.15.32:0/3818413460 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7788038040 con 0x7f77980ce3e0 2024-06-09T22:01:04.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.983+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77981523b0 con 0x7f77980ce3e0 2024-06-09T22:01:04.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.983+0000 7f778dffb640 1 -- 172.21.15.32:0/3818413460 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f778803cb50 con 0x7f77980ce3e0 2024-06-09T22:01:04.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.983+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7798152970 con 0x7f77980ce3e0 2024-06-09T22:01:04.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:04.984+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7798068490 con 0x7f77980ce3e0 2024-06-09T22:01:05.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.270+0000 7f778dffb640 1 -- 172.21.15.32:0/3818413460 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f778803ccf0 con 0x7f77980ce3e0 2024-06-09T22:01:05.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.270+0000 7f778dffb640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f777c06be80 0x7f777c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:05.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.270+0000 7f779d236640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f777c06be80 0x7f777c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:05.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.270+0000 7f779d236640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f777c06be80 0x7f777c06e340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7794005880 tx=0x7f7794009040 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.287+0000 7f778dffb640 1 -- 172.21.15.32:0/3818413460 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f7788078670 con 0x7f77980ce3e0 2024-06-09T22:01:05.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.287+0000 7f778dffb640 1 -- 172.21.15.32:0/3818413460 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f778807a5d0 con 0x7f77980ce3e0 2024-06-09T22:01:05.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.489+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f779806e120 con 0x7f77980ce3e0 2024-06-09T22:01:05.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.629+0000 7f778dffb640 1 -- 172.21.15.32:0/3818413460 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18506 (secure 0 0 0) 0x7f778807e8d0 con 0x7f77980ce3e0 2024-06-09T22:01:05.614 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:01:05.614 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"fsid":"08d1d92e-3533-4051-bd41-97c73d8b6b19","created":"2024-06-09T22:00:27.411675+0000","modified":"2024-06-09T22:01:03.716360+0000","last_up_change":"2024-06-09T22:00:42.024015+0000","last_in_change":"2024-06-09T22:00:33.132294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:00:41.893545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:00:54.102789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:00:55.454518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:01:00.813566+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"4cc9d666-a633-4740-a293-be07ab208937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6811","nonce":3275688150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6813","nonce":3275688150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6817","nonce":3275688150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6815","nonce":3275688150}]},"public_addr":"172.21.15.32:6811/3275688150","cluster_addr":"172.21.15.32:6813/3275688150","heartbeat_back_addr":"172.21.15.32:6817/3275688150","heartbeat_front_addr":"172.21.15.32:6815/3275688150","state":["exists","up"]},{"osd":1,"uuid":"8db4805f-af3c-41a5-8eaf-1485c4ec8a5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6803","nonce":3057205177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6805","nonce":3057205177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6809","nonce":3057205177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6807","nonce":3057205177}]},"public_addr":"172.21.15.32:6803/3057205177","cluster_addr":"172.21.15.32:6805/3057205177","heartbeat_back_addr":"172.21.15.32:6809/3057205177","heartbeat_front_addr":"172.21.15.32:6807/3057205177","state":["exists","up"]},{"osd":2,"uuid":"172c9719-1b63-460d-ba3e-dba05f194990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6819","nonce":3248807238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6821","nonce":3248807238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6825","nonce":3248807238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6823","nonce":3248807238}]},"public_addr":"172.21.15.32:6819/3248807238","cluster_addr":"172.21.15.32:6821/3248807238","heartbeat_back_addr":"172.21.15.32:6825/3248807238","heartbeat_front_addr":"172.21.15.32:6823/3248807238","state":["exists","up"]},{"osd":3,"uuid":"611e5921-1524-4cc9-9f33-a51d431115f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6827","nonce":1835052153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6829","nonce":1835052153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6833","nonce":1835052153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6831","nonce":1835052153}]},"public_addr":"172.21.15.32:6827/1835052153","cluster_addr":"172.21.15.32:6829/1835052153","heartbeat_back_addr":"172.21.15.32:6833/1835052153","heartbeat_front_addr":"172.21.15.32:6831/1835052153","state":["exists","up"]},{"osd":4,"uuid":"b85f7437-19ea-4f08-8a71-ef14330e0351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6801","nonce":531504437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6803","nonce":531504437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6807","nonce":531504437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6805","nonce":531504437}]},"public_addr":"172.21.15.148:6801/531504437","cluster_addr":"172.21.15.148:6803/531504437","heartbeat_back_addr":"172.21.15.148:6807/531504437","heartbeat_front_addr":"172.21.15.148:6805/531504437","state":["exists","up"]},{"osd":5,"uuid":"ae110414-eb0e-4253-91f2-0b2e83dad115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6810","nonce":2398614348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6814","nonce":2398614348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6822","nonce":2398614348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6818","nonce":2398614348}]},"public_addr":"172.21.15.148:6810/2398614348","cluster_addr":"172.21.15.148:6814/2398614348","heartbeat_back_addr":"172.21.15.148:6822/2398614348","heartbeat_front_addr":"172.21.15.148:6818/2398614348","state":["exists","up"]},{"osd":6,"uuid":"0de55946-8987-4376-b9f9-f4957200232f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6811","nonce":975384648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6815","nonce":975384648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6821","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6823","nonce":975384648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6817","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6819","nonce":975384648}]},"public_addr":"172.21.15.148:6811/975384648","cluster_addr":"172.21.15.148:6815/975384648","heartbeat_back_addr":"172.21.15.148:6823/975384648","heartbeat_front_addr":"172.21.15.148:6819/975384648","state":["exists","up"]},{"osd":7,"uuid":"0524e1f0-9c95-4d96-8d3f-e120b647bc83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6825","nonce":2655085429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6827","nonce":2655085429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6831","nonce":2655085429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6829","nonce":2655085429}]},"public_addr":"172.21.15.148:6825/2655085429","cluster_addr":"172.21.15.148:6827/2655085429","heartbeat_back_addr":"172.21.15.148:6831/2655085429","heartbeat_front_addr":"172.21.15.148:6829/2655085429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.620927+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.493643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.657004+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.616764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.596137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.580349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.602447+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.609775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:01:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f777c06be80 msgr2=0x7f777c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f777c06be80 0x7f777c06e340 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7794005880 tx=0x7f7794009040 comp rx=0 tx=0).stop 2024-06-09T22:01:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77980ce3e0 msgr2=0x7f7798153a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77980ce3e0 0x7f7798153a30 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f77880098f0 tx=0x7f77880046c0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 shutdown_connections 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f777c06be80 0x7f777c06e340 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f77980cedc0 0x7f77980c8870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77980ce3e0 0x7f7798153a30 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 --2- 172.21.15.32:0/3818413460 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f77980c95e0 0x7f77981f0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 >> 172.21.15.32:0/3818413460 conn(0x7f77980b18d0 msgr2=0x7f7798159200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 shutdown_connections 2024-06-09T22:01:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.631+0000 7f779f49a640 1 -- 172.21.15.32:0/3818413460 wait complete. 2024-06-09T22:01:05.626 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-09T22:01:05.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.745+0000 7f2b4f831640 1 Processor -- start 2024-06-09T22:01:05.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.745+0000 7f2b4f831640 1 -- start start 2024-06-09T22:01:05.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.746+0000 7f2b4f831640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b48154420 0x7f2b480c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:05.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.746+0000 7f2b4f831640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7f2b4815e520 0x7f2b4815d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:05.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.746+0000 7f2b4f831640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b480cc920 0x7f2b4815def0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:05.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.746+0000 7f2b4f831640 1 -- --> v2:172.21.15.32:3300/0 -- mon_getmap magic: 0 -- 0x7f2b480cbc00 con 0x7f2b480cc920 2024-06-09T22:01:05.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.746+0000 7f2b4f831640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7f2b4806d130 con 0x7f2b4815e520 2024-06-09T22:01:05.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.746+0000 7f2b4f831640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f2b480cbe60 con 0x7f2b48154420 2024-06-09T22:01:05.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.746+0000 7f2b4ddce640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b480cc920 0x7f2b4815def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:05.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.747+0000 7f2b4d5cd640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b48154420 0x7f2b480c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:05.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.747+0000 7f2b4ddce640 1 --2- >> v2:172.21.15.32:3300/0 conn(0x7f2b480cc920 0x7f2b4815def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59804/0 (socket says 172.21.15.32:59804) 2024-06-09T22:01:05.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.747+0000 7f2b4ddce640 1 -- 172.21.15.32:0/3897680147 learned_addr learned my addr 172.21.15.32:0/3897680147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:01:05.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.747+0000 7f2b4cdcc640 1 --2- 172.21.15.32:0/3897680147 >> v2:172.21.15.148:3300/0 conn(0x7f2b4815e520 0x7f2b4815d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:05.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.749+0000 7f2b4ddce640 1 -- 172.21.15.32:0/3897680147 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b48154420 msgr2=0x7f2b480c96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:05.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.749+0000 7f2b4ddce640 1 --2- 172.21.15.32:0/3897680147 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b48154420 0x7f2b480c96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.749+0000 7f2b4ddce640 1 -- 172.21.15.32:0/3897680147 >> v2:172.21.15.148:3300/0 conn(0x7f2b4815e520 msgr2=0x7f2b4815d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:05.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.749+0000 7f2b4ddce640 1 --2- 172.21.15.32:0/3897680147 >> v2:172.21.15.148:3300/0 conn(0x7f2b4815e520 0x7f2b4815d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.749+0000 7f2b4ddce640 1 -- 172.21.15.32:0/3897680147 --> v2:172.21.15.32:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b480cc120 con 0x7f2b480cc920 2024-06-09T22:01:05.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.751+0000 7f2b4ddce640 1 --2- 172.21.15.32:0/3897680147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b480cc920 0x7f2b4815def0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2b4806ef10 tx=0x7f2b4002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1504a869f58743da server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:05.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.755+0000 7f2b3ffff640 1 -- 172.21.15.32:0/3897680147 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b40036070 con 0x7f2b480cc920 2024-06-09T22:01:05.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.755+0000 7f2b3ffff640 1 -- 172.21.15.32:0/3897680147 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b40031600 con 0x7f2b480cc920 2024-06-09T22:01:05.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.755+0000 7f2b3ffff640 1 -- 172.21.15.32:0/3897680147 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b400319d0 con 0x7f2b480cc920 2024-06-09T22:01:05.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 -- 172.21.15.32:0/3897680147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b480cc920 msgr2=0x7f2b4815def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:05.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 --2- 172.21.15.32:0/3897680147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b480cc920 0x7f2b4815def0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2b4806ef10 tx=0x7f2b4002a6a0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 -- 172.21.15.32:0/3897680147 shutdown_connections 2024-06-09T22:01:05.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 --2- 172.21.15.32:0/3897680147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b480cc920 0x7f2b4815def0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 --2- 172.21.15.32:0/3897680147 >> v2:172.21.15.148:3300/0 conn(0x7f2b4815e520 0x7f2b4815d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 --2- 172.21.15.32:0/3897680147 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b48154420 0x7f2b480c96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 -- 172.21.15.32:0/3897680147 >> 172.21.15.32:0/3897680147 conn(0x7f2b480717f0 msgr2=0x7f2b48071bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:05.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 -- 172.21.15.32:0/3897680147 shutdown_connections 2024-06-09T22:01:05.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.756+0000 7f2b4f831640 1 -- 172.21.15.32:0/3897680147 wait complete. 2024-06-09T22:01:05.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.757+0000 7f2b4f831640 1 Processor -- start 2024-06-09T22:01:05.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.757+0000 7f2b4f831640 1 -- start start 2024-06-09T22:01:05.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.758+0000 7f2b4f831640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b480cc920 0x7f2b4814efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:05.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.758+0000 7f2b4f831640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b48154420 0x7f2b480b2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:05.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.758+0000 7f2b4f831640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815e520 0x7f2b480b27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:05.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.758+0000 7f2b4f831640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2b4806d3d0 con 0x7f2b48154420 2024-06-09T22:01:05.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.758+0000 7f2b4f831640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7f2b48154800 con 0x7f2b4815e520 2024-06-09T22:01:05.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.758+0000 7f2b4f831640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7f2b4806c4a0 con 0x7f2b480cc920 2024-06-09T22:01:05.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.759+0000 7f2b4cdcc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b48154420 0x7f2b480b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:05.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.759+0000 7f2b4cdcc640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b48154420 0x7f2b480b2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59816/0 (socket says 172.21.15.32:59816) 2024-06-09T22:01:05.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.759+0000 7f2b4cdcc640 1 -- 172.21.15.32:0/2094786393 learned_addr learned my addr 172.21.15.32:0/2094786393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:01:05.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.759+0000 7f2b4ddce640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815e520 0x7f2b480b27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:05.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.759+0000 7f2b4d5cd640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b480cc920 0x7f2b4814efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:05.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.763+0000 7f2b4cdcc640 1 -- 172.21.15.32:0/2094786393 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b480cc920 msgr2=0x7f2b4814efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:05.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.763+0000 7f2b4cdcc640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b480cc920 0x7f2b4814efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.763+0000 7f2b4cdcc640 1 -- 172.21.15.32:0/2094786393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815e520 msgr2=0x7f2b480b27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:05.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.763+0000 7f2b4cdcc640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815e520 0x7f2b480b27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:05.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.763+0000 7f2b4cdcc640 1 -- 172.21.15.32:0/2094786393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b4814eb00 con 0x7f2b48154420 2024-06-09T22:01:05.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.765+0000 7f2b4cdcc640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b48154420 0x7f2b480b2260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800e870 tx=0x7f2b3800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:05.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.768+0000 7f2b3dffb640 1 -- 172.21.15.32:0/2094786393 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b38017070 con 0x7f2b48154420 2024-06-09T22:01:05.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.768+0000 7f2b3dffb640 1 -- 172.21.15.32:0/2094786393 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2b3800c940 con 0x7f2b48154420 2024-06-09T22:01:05.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.768+0000 7f2b3dffb640 1 -- 172.21.15.32:0/2094786393 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b3800cd30 con 0x7f2b48154420 2024-06-09T22:01:05.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.768+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b4814f9f0 con 0x7f2b48154420 2024-06-09T22:01:05.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.769+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b48150040 con 0x7f2b48154420 2024-06-09T22:01:05.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:05.769+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b4806e510 con 0x7f2b48154420 2024-06-09T22:01:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.056+0000 7f2b3dffb640 1 -- 172.21.15.32:0/2094786393 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7f2b3801a070 con 0x7f2b48154420 2024-06-09T22:01:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.056+0000 7f2b3dffb640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2b2c06be80 0x7f2b2c06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.056+0000 7f2b4d5cd640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2b2c06be80 0x7f2b2c06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:06.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.057+0000 7f2b4d5cd640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2b2c06be80 0x7f2b2c06e340 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b44009b40 tx=0x7f2b44009190 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:06.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.070+0000 7f2b3dffb640 1 -- 172.21.15.32:0/2094786393 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7f2b380577b0 con 0x7f2b48154420 2024-06-09T22:01:06.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.070+0000 7f2b3dffb640 1 -- 172.21.15.32:0/2094786393 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7f2b4806e510 con 0x7f2b48154420 2024-06-09T22:01:06.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.257+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b48068490 con 0x7f2b48154420 2024-06-09T22:01:06.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.300+0000 7f2b3dffb640 1 -- 172.21.15.32:0/2094786393 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18506 (secure 0 0 0) 0x7f2b38059710 con 0x7f2b48154420 2024-06-09T22:01:06.284 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:01:06.284 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"fsid":"08d1d92e-3533-4051-bd41-97c73d8b6b19","created":"2024-06-09T22:00:27.411675+0000","modified":"2024-06-09T22:01:03.716360+0000","last_up_change":"2024-06-09T22:00:42.024015+0000","last_in_change":"2024-06-09T22:00:33.132294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:00:41.893545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:00:54.102789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:00:55.454518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:01:00.813566+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"4cc9d666-a633-4740-a293-be07ab208937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6811","nonce":3275688150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6813","nonce":3275688150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6817","nonce":3275688150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6815","nonce":3275688150}]},"public_addr":"172.21.15.32:6811/3275688150","cluster_addr":"172.21.15.32:6813/3275688150","heartbeat_back_addr":"172.21.15.32:6817/3275688150","heartbeat_front_addr":"172.21.15.32:6815/3275688150","state":["exists","up"]},{"osd":1,"uuid":"8db4805f-af3c-41a5-8eaf-1485c4ec8a5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6803","nonce":3057205177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6805","nonce":3057205177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6809","nonce":3057205177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6807","nonce":3057205177}]},"public_addr":"172.21.15.32:6803/3057205177","cluster_addr":"172.21.15.32:6805/3057205177","heartbeat_back_addr":"172.21.15.32:6809/3057205177","heartbeat_front_addr":"172.21.15.32:6807/3057205177","state":["exists","up"]},{"osd":2,"uuid":"172c9719-1b63-460d-ba3e-dba05f194990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6819","nonce":3248807238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6821","nonce":3248807238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6825","nonce":3248807238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6823","nonce":3248807238}]},"public_addr":"172.21.15.32:6819/3248807238","cluster_addr":"172.21.15.32:6821/3248807238","heartbeat_back_addr":"172.21.15.32:6825/3248807238","heartbeat_front_addr":"172.21.15.32:6823/3248807238","state":["exists","up"]},{"osd":3,"uuid":"611e5921-1524-4cc9-9f33-a51d431115f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6827","nonce":1835052153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6829","nonce":1835052153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6833","nonce":1835052153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6831","nonce":1835052153}]},"public_addr":"172.21.15.32:6827/1835052153","cluster_addr":"172.21.15.32:6829/1835052153","heartbeat_back_addr":"172.21.15.32:6833/1835052153","heartbeat_front_addr":"172.21.15.32:6831/1835052153","state":["exists","up"]},{"osd":4,"uuid":"b85f7437-19ea-4f08-8a71-ef14330e0351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6801","nonce":531504437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6803","nonce":531504437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6807","nonce":531504437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6805","nonce":531504437}]},"public_addr":"172.21.15.148:6801/531504437","cluster_addr":"172.21.15.148:6803/531504437","heartbeat_back_addr":"172.21.15.148:6807/531504437","heartbeat_front_addr":"172.21.15.148:6805/531504437","state":["exists","up"]},{"osd":5,"uuid":"ae110414-eb0e-4253-91f2-0b2e83dad115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6810","nonce":2398614348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6814","nonce":2398614348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6822","nonce":2398614348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6818","nonce":2398614348}]},"public_addr":"172.21.15.148:6810/2398614348","cluster_addr":"172.21.15.148:6814/2398614348","heartbeat_back_addr":"172.21.15.148:6822/2398614348","heartbeat_front_addr":"172.21.15.148:6818/2398614348","state":["exists","up"]},{"osd":6,"uuid":"0de55946-8987-4376-b9f9-f4957200232f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6811","nonce":975384648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6815","nonce":975384648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6821","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6823","nonce":975384648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6817","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6819","nonce":975384648}]},"public_addr":"172.21.15.148:6811/975384648","cluster_addr":"172.21.15.148:6815/975384648","heartbeat_back_addr":"172.21.15.148:6823/975384648","heartbeat_front_addr":"172.21.15.148:6819/975384648","state":["exists","up"]},{"osd":7,"uuid":"0524e1f0-9c95-4d96-8d3f-e120b647bc83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6825","nonce":2655085429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6827","nonce":2655085429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6831","nonce":2655085429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6829","nonce":2655085429}]},"public_addr":"172.21.15.148:6825/2655085429","cluster_addr":"172.21.15.148:6827/2655085429","heartbeat_back_addr":"172.21.15.148:6831/2655085429","heartbeat_front_addr":"172.21.15.148:6829/2655085429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.620927+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.493643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.657004+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.616764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.596137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.580349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.602447+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.609775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:01:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2b2c06be80 msgr2=0x7f2b2c06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:06.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2b2c06be80 0x7f2b2c06e340 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2b44009b40 tx=0x7f2b44009190 comp rx=0 tx=0).stop 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b48154420 msgr2=0x7f2b480b2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b48154420 0x7f2b480b2260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2b3800e870 tx=0x7f2b3800ed30 comp rx=0 tx=0).stop 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 shutdown_connections 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7f2b4815e520 0x7f2b480b27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2b48154420 0x7f2b480b2260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7f2b2c06be80 0x7f2b2c06e340 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 --2- 172.21.15.32:0/2094786393 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7f2b480cc920 0x7f2b4814efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 >> 172.21.15.32:0/2094786393 conn(0x7f2b480717f0 msgr2=0x7f2b48152870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 shutdown_connections 2024-06-09T22:01:06.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.302+0000 7f2b4f831640 1 -- 172.21.15.32:0/2094786393 wait complete. 2024-06-09T22:01:06.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-06-09T22:01:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.434+0000 7fce893d5640 1 Processor -- start 2024-06-09T22:01:06.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.434+0000 7fce893d5640 1 -- start start 2024-06-09T22:01:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.435+0000 7fce893d5640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fce84154810 0x7fce84073100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.435+0000 7fce893d5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce840705d0 0x7fce8406b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.435+0000 7fce893d5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce840c7870 con 0x7fce84154430 2024-06-09T22:01:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.435+0000 7fce893d5640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7fce840c7380 con 0x7fce84154810 2024-06-09T22:01:06.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.435+0000 7fce893d5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fce840c8b40 con 0x7fce840705d0 2024-06-09T22:01:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.436+0000 7fce82ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fce84154430 0x7fce8415d900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33182/0 (socket says 172.21.15.32:33182) 2024-06-09T22:01:06.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.436+0000 7fce82ffd640 1 -- 172.21.15.32:0/3429467563 learned_addr learned my addr 172.21.15.32:0/3429467563 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:01:06.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.438+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1905475764 0 0) 0x7fce840c7870 con 0x7fce84154430 2024-06-09T22:01:06.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.438+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce70003520 con 0x7fce84154430 2024-06-09T22:01:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3352052440 0 0) 0x7fce70003520 con 0x7fce84154430 2024-06-09T22:01:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce840705d0 msgr2=0x7fce8406b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce840705d0 0x7fce8406b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 >> v2:172.21.15.148:3300/0 conn(0x7fce84154810 msgr2=0x7fce84073100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 --2- 172.21.15.32:0/3429467563 >> v2:172.21.15.148:3300/0 conn(0x7fce84154810 0x7fce84073100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce840c75b0 con 0x7fce84154430 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 909852050 0 0) 0x7fce740019f0 con 0x7fce84154430 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 >> v1:172.21.15.32:6789/0 conn(0x7fce84154430 legacy=0x7fce8415d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce840705d0 0x7fce700039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce700040c0 0x7fce70004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce70004d90 0x7fce70005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fce840c8b40 con 0x7fce700040c0 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fce7000bc00 con 0x7fce70004d90 2024-06-09T22:01:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.442+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fce7000be00 con 0x7fce840705d0 2024-06-09T22:01:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.443+0000 7fce827fc640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce700040c0 0x7fce70004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:06.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.448+0000 7fce827fc640 1 -- 172.21.15.32:0/3429467563 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce840705d0 msgr2=0x7fce700039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:06.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.448+0000 7fce827fc640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce840705d0 0x7fce700039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.448+0000 7fce827fc640 1 -- 172.21.15.32:0/3429467563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce70004d90 msgr2=0x7fce70005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:06.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.448+0000 7fce827fc640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce70004d90 0x7fce70005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.448+0000 7fce827fc640 1 -- 172.21.15.32:0/3429467563 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce70003520 con 0x7fce700040c0 2024-06-09T22:01:06.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.451+0000 7fce827fc640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce700040c0 0x7fce70004800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009860 tx=0x7fce70005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9d580c2f89ed3c5 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:06.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.455+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c037070 con 0x7fce700040c0 2024-06-09T22:01:06.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.455+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce6c02aaa0 con 0x7fce700040c0 2024-06-09T22:01:06.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.455+0000 7fce81ffb640 1 -- 172.21.15.32:0/3429467563 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c009460 con 0x7fce700040c0 2024-06-09T22:01:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.456+0000 7fce893d5640 1 -- 172.21.15.32:0/3429467563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce700040c0 msgr2=0x7fce70004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.456+0000 7fce893d5640 1 --2- 172.21.15.32:0/3429467563 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce700040c0 0x7fce70004800 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce6c009860 tx=0x7fce70005830 comp rx=0 tx=0).stop 2024-06-09T22:01:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.456+0000 7fce837fe640 1 -- 172.21.15.32:0/3429467563 reap_dead start 2024-06-09T22:01:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.456+0000 7fce893d5640 1 -- 172.21.15.32:0/3429467563 shutdown_connections 2024-06-09T22:01:06.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.456+0000 7fce893d5640 1 -- 172.21.15.32:0/3429467563 >> 172.21.15.32:0/3429467563 conn(0x7fce84072270 msgr2=0x7fce84072680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:06.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.456+0000 7fce893d5640 1 -- 172.21.15.32:0/3429467563 shutdown_connections 2024-06-09T22:01:06.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.456+0000 7fce893d5640 1 -- 172.21.15.32:0/3429467563 wait complete. 2024-06-09T22:01:06.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.458+0000 7fce893d5640 1 Processor -- start 2024-06-09T22:01:06.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.458+0000 7fce893d5640 1 -- start start 2024-06-09T22:01:06.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.459+0000 7fce893d5640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce8406ba80 0x7fce8414d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.459+0000 7fce893d5640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8406be60 0x7fce8406f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.459+0000 7fce893d5640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce840ca020 0x7fce8406f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.459+0000 7fce893d5640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fce8406b390 con 0x7fce840ca020 2024-06-09T22:01:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.459+0000 7fce893d5640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fce840c8860 con 0x7fce8406be60 2024-06-09T22:01:06.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.459+0000 7fce893d5640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fce8406b510 con 0x7fce8406ba80 2024-06-09T22:01:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.459+0000 7fce827fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8406be60 0x7fce8406f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.460+0000 7fce827fc640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8406be60 0x7fce8406f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.148:3300/0 says I am v2:172.21.15.32:39744/0 (socket says 172.21.15.32:39744) 2024-06-09T22:01:06.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.460+0000 7fce827fc640 1 -- 172.21.15.32:0/3358558306 learned_addr learned my addr 172.21.15.32:0/3358558306 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:01:06.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.462+0000 7fce82ffd640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce8406ba80 0x7fce8414d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:06.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.462+0000 7fce837fe640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce840ca020 0x7fce8406f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.464+0000 7fce827fc640 1 -- 172.21.15.32:0/3358558306 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce8406ba80 msgr2=0x7fce8414d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.464+0000 7fce827fc640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce8406ba80 0x7fce8414d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.464+0000 7fce827fc640 1 -- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce840ca020 msgr2=0x7fce8406f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.464+0000 7fce827fc640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce840ca020 0x7fce8406f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:06.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.464+0000 7fce827fc640 1 -- 172.21.15.32:0/3358558306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce841e5ab0 con 0x7fce8406be60 2024-06-09T22:01:06.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.466+0000 7fce827fc640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8406be60 0x7fce8406f2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fce6c02a530 tx=0x7fce6c0047d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.470+0000 7fce7b7fe640 1 -- 172.21.15.32:0/3358558306 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c037040 con 0x7fce8406be60 2024-06-09T22:01:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.470+0000 7fce7b7fe640 1 -- 172.21.15.32:0/3358558306 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce6c0037f0 con 0x7fce8406be60 2024-06-09T22:01:06.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.470+0000 7fce7b7fe640 1 -- 172.21.15.32:0/3358558306 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce6c003ba0 con 0x7fce8406be60 2024-06-09T22:01:06.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.470+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce841e5d90 con 0x7fce8406be60 2024-06-09T22:01:06.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.470+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce840cbd90 con 0x7fce8406be60 2024-06-09T22:01:06.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.471+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce840c5c90 con 0x7fce8406be60 2024-06-09T22:01:06.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.492+0000 7fce7b7fe640 1 -- 172.21.15.32:0/3358558306 <== mon.1 v2:172.21.15.148:3300/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fce6c03d470 con 0x7fce8406be60 2024-06-09T22:01:06.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.493+0000 7fce7b7fe640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fce6406be80 0x7fce6406e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:06.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.493+0000 7fce82ffd640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fce6406be80 0x7fce6406e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:06.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.493+0000 7fce82ffd640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fce6406be80 0x7fce6406e340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fce74001330 tx=0x7fce74001de0 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:06.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.496+0000 7fce7b7fe640 1 -- 172.21.15.32:0/3358558306 <== mon.1 v2:172.21.15.148:3300/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fce6c077f30 con 0x7fce8406be60 2024-06-09T22:01:06.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:06.803+0000 7fce7b7fe640 1 -- 172.21.15.32:0/3358558306 <== mon.1 v2:172.21.15.148:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7fce840c5c90 con 0x7fce8406be60 2024-06-09T22:01:07.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.026+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce84068490 con 0x7fce8406be60 2024-06-09T22:01:07.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.042+0000 7fce7b7fe640 1 -- 172.21.15.32:0/3358558306 <== mon.1 v2:172.21.15.148:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+6834 (secure 0 0 0) 0x7fce6c079e90 con 0x7fce8406be60 2024-06-09T22:01:07.026 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:01:07.026 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4199,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2870873470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2870873470},{"type":"v1","addr":"172.21.15.32:6835","nonce":2870873470}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4240,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6833/1513914407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6832","nonce":1513914407},{"type":"v1","addr":"172.21.15.148:6833","nonce":1513914407}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4243,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6835/2699555510","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6834","nonce":2699555510},{"type":"v1","addr":"172.21.15.148:6835","nonce":2699555510}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4257,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.148:6837/2047518031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6836","nonce":2047518031},{"type":"v1","addr":"172.21.15.148:6837","nonce":2047518031}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4260,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2947563635","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2947563635},{"type":"v1","addr":"172.21.15.32:6837","nonce":2947563635}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-06-09T22:00:56.720976+0000","modified":"2024-06-09T22:01:03.721677+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4263},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4263":{"gid":4263,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.32:6839/3734386334","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":3734386334},{"type":"v1","addr":"172.21.15.32:6839","nonce":3734386334}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":4263,"qdb_cluster":[4263]},"id":1}]} 2024-06-09T22:01:07.026 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 7 2024-06-09T22:01:07.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fce6406be80 msgr2=0x7fce6406e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fce6406be80 0x7fce6406e340 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fce74001330 tx=0x7fce74001de0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8406be60 msgr2=0x7fce8406f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8406be60 0x7fce8406f2d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fce6c02a530 tx=0x7fce6c0047d0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 shutdown_connections 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce840ca020 0x7fce8406f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fce8406be60 0x7fce8406f2d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fce6406be80 0x7fce6406e340 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 --2- 172.21.15.32:0/3358558306 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fce8406ba80 0x7fce8414d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 >> 172.21.15.32:0/3358558306 conn(0x7fce84072270 msgr2=0x7fce84151e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 shutdown_connections 2024-06-09T22:01:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.044+0000 7fce893d5640 1 -- 172.21.15.32:0/3358558306 wait complete. 2024-06-09T22:01:07.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-09T22:01:07.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.162+0000 7fe4d2329640 1 Processor -- start 2024-06-09T22:01:07.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.162+0000 7fe4d2329640 1 -- start start 2024-06-09T22:01:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.162+0000 7fe4d2329640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc150d70 0x7fe4cc0c0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.163+0000 7fe4d2329640 1 --2- >> v2:172.21.15.148:3300/0 conn(0x7fe4cc157690 0x7fe4cc071a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.163+0000 7fe4d2329640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4cc0c1410 con 0x7fe4cc073590 2024-06-09T22:01:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.163+0000 7fe4d2329640 1 -- --> v2:172.21.15.148:3300/0 -- mon_getmap magic: 0 -- 0x7fe4cc06fb10 con 0x7fe4cc157690 2024-06-09T22:01:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.163+0000 7fe4d2329640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fe4cc151150 con 0x7fe4cc150d70 2024-06-09T22:01:07.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.163+0000 7fe4d08c6640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe4cc073590 0x7fe4cc0b1750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:33186/0 (socket says 172.21.15.32:33186) 2024-06-09T22:01:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.163+0000 7fe4cbfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc150d70 0x7fe4cc0c0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.163+0000 7fe4d08c6640 1 -- 172.21.15.32:0/4289172867 learned_addr learned my addr 172.21.15.32:0/4289172867 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-06-09T22:01:07.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.163+0000 7fe4c3fff640 1 --2- 172.21.15.32:0/4289172867 >> v2:172.21.15.148:3300/0 conn(0x7fe4cc157690 0x7fe4cc071a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.166+0000 7fe4c3fff640 1 -- 172.21.15.32:0/4289172867 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc150d70 msgr2=0x7fe4cc0c0ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-09T22:01:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.166+0000 7fe4cb7fe640 1 -- 172.21.15.32:0/4289172867 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1853149370 0 0) 0x7fe4cc0c1410 con 0x7fe4cc073590 2024-06-09T22:01:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.166+0000 7fe4c3fff640 1 --2- 172.21.15.32:0/4289172867 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc150d70 0x7fe4cc0c0ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.166+0000 7fe4c3fff640 1 -- 172.21.15.32:0/4289172867 >> v1:172.21.15.32:6789/0 conn(0x7fe4cc073590 legacy=0x7fe4cc0b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.166+0000 7fe4c3fff640 1 -- 172.21.15.32:0/4289172867 --> v2:172.21.15.148:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4cc06f550 con 0x7fe4cc157690 2024-06-09T22:01:07.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.166+0000 7fe4cbfff640 1 --2- 172.21.15.32:0/4289172867 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc150d70 0x7fe4cc0c0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:01:07.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.168+0000 7fe4c3fff640 1 --2- 172.21.15.32:0/4289172867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 0x7fe4cc071a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40097a0 tx=0x7fe4b402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aad9989fbc02790b server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:07.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.171+0000 7fe4cb7fe640 1 -- 172.21.15.32:0/4289172867 <== mon.1 v2:172.21.15.148:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4037070 con 0x7fe4cc157690 2024-06-09T22:01:07.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.171+0000 7fe4cb7fe640 1 -- 172.21.15.32:0/4289172867 <== mon.1 v2:172.21.15.148:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b4032440 con 0x7fe4cc157690 2024-06-09T22:01:07.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.171+0000 7fe4cb7fe640 1 -- 172.21.15.32:0/4289172867 <== mon.1 v2:172.21.15.148:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b4032810 con 0x7fe4cc157690 2024-06-09T22:01:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.172+0000 7fe4d2329640 1 -- 172.21.15.32:0/4289172867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 msgr2=0x7fe4cc071a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.172+0000 7fe4d2329640 1 --2- 172.21.15.32:0/4289172867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 0x7fe4cc071a50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fe4b40097a0 tx=0x7fe4b402a210 comp rx=0 tx=0).stop 2024-06-09T22:01:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.172+0000 7fe4d2329640 1 -- 172.21.15.32:0/4289172867 shutdown_connections 2024-06-09T22:01:07.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.172+0000 7fe4d2329640 1 --2- 172.21.15.32:0/4289172867 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 0x7fe4cc071a50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.172+0000 7fe4d2329640 1 --2- 172.21.15.32:0/4289172867 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc150d70 0x7fe4cc0c0ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.172+0000 7fe4d2329640 1 -- 172.21.15.32:0/4289172867 >> 172.21.15.32:0/4289172867 conn(0x7fe4cc0cbb70 msgr2=0x7fe4cc0cbf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.172+0000 7fe4d2329640 1 -- 172.21.15.32:0/4289172867 shutdown_connections 2024-06-09T22:01:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.172+0000 7fe4d2329640 1 -- 172.21.15.32:0/4289172867 wait complete. 2024-06-09T22:01:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.173+0000 7fe4d2329640 1 Processor -- start 2024-06-09T22:01:07.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.173+0000 7fe4d2329640 1 -- start start 2024-06-09T22:01:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.174+0000 7fe4d2329640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc073590 0x7fe4cc140360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.174+0000 7fe4d2329640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4cc150d70 0x7fe4cc0c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.174+0000 7fe4d2329640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 0x7fe4cc0c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.174+0000 7fe4d2329640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe4cc156e90 con 0x7fe4cc150d70 2024-06-09T22:01:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.174+0000 7fe4d2329640 1 -- --> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] -- mon_getmap magic: 0 -- 0x7fe4cc156bb0 con 0x7fe4cc157690 2024-06-09T22:01:07.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.174+0000 7fe4d2329640 1 -- --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_getmap magic: 0 -- 0x7fe4cc157170 con 0x7fe4cc073590 2024-06-09T22:01:07.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.175+0000 7fe4c3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4cc150d70 0x7fe4cc0c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:07.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.175+0000 7fe4d08c6640 1 --2- >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 0x7fe4cc0c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:07.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.175+0000 7fe4cbfff640 1 --2- >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc073590 0x7fe4cc140360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:07.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.175+0000 7fe4c3fff640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4cc150d70 0x7fe4cc0c36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59850/0 (socket says 172.21.15.32:59850) 2024-06-09T22:01:07.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.175+0000 7fe4c3fff640 1 -- 172.21.15.32:0/2155546528 learned_addr learned my addr 172.21.15.32:0/2155546528 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-06-09T22:01:07.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.179+0000 7fe4cbfff640 1 -- 172.21.15.32:0/2155546528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 msgr2=0x7fe4cc0c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:07.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.179+0000 7fe4cbfff640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 0x7fe4cc0c3c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.179+0000 7fe4cbfff640 1 -- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4cc150d70 msgr2=0x7fe4cc0c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:07.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.179+0000 7fe4cbfff640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4cc150d70 0x7fe4cc0c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.179+0000 7fe4cbfff640 1 -- 172.21.15.32:0/2155546528 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4cc1529c0 con 0x7fe4cc073590 2024-06-09T22:01:07.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.179+0000 7fe4c3fff640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4cc150d70 0x7fe4cc0c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-09T22:01:07.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.181+0000 7fe4cbfff640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc073590 0x7fe4cc140360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00e930 tx=0x7fe4bc00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:07.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.185+0000 7fe4c97fa640 1 -- 172.21.15.32:0/2155546528 <== mon.2 v2:172.21.15.148:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc017070 con 0x7fe4cc073590 2024-06-09T22:01:07.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.185+0000 7fe4c97fa640 1 -- 172.21.15.32:0/2155546528 <== mon.2 v2:172.21.15.148:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc00ca00 con 0x7fe4cc073590 2024-06-09T22:01:07.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.185+0000 7fe4c97fa640 1 -- 172.21.15.32:0/2155546528 <== mon.2 v2:172.21.15.148:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc00cdd0 con 0x7fe4cc073590 2024-06-09T22:01:07.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.185+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4cc0c5100 con 0x7fe4cc073590 2024-06-09T22:01:07.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.185+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4cc1f2180 con 0x7fe4cc073590 2024-06-09T22:01:07.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.186+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4cc0c58d0 con 0x7fe4cc073590 2024-06-09T22:01:07.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.205+0000 7fe4c97fa640 1 -- 172.21.15.32:0/2155546528 <== mon.2 v2:172.21.15.148:3301/0 4 ==== mgrmap(e 5) ==== 90498+0+0 (secure 0 0 0) 0x7fe4bc01a070 con 0x7fe4cc073590 2024-06-09T22:01:07.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.205+0000 7fe4c97fa640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe4ac06be80 0x7fe4ac06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-09T22:01:07.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.205+0000 7fe4c3fff640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe4ac06be80 0x7fe4ac06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-09T22:01:07.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.206+0000 7fe4c3fff640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe4ac06be80 0x7fe4ac06e340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc1528b0 tx=0x7fe4b402e000 comp rx=0 tx=0).ready entity=mgr.4105 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-09T22:01:07.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.208+0000 7fe4c97fa640 1 -- 172.21.15.32:0/2155546528 <== mon.2 v2:172.21.15.148:3301/0 5 ==== osd_map(26..26 src has 1..26) ==== 6532+0+0 (secure 0 0 0) 0x7fe4bc057860 con 0x7fe4cc073590 2024-06-09T22:01:07.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.452+0000 7fe4c97fa640 1 -- 172.21.15.32:0/2155546528 <== mon.2 v2:172.21.15.148:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+185294 (secure 0 0 0) 0x7fe4cc0c58d0 con 0x7fe4cc073590 2024-06-09T22:01:07.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.644+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 --> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe4cc13d430 con 0x7fe4cc073590 2024-06-09T22:01:07.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.706+0000 7fe4c97fa640 1 -- 172.21.15.32:0/2155546528 <== mon.2 v2:172.21.15.148:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v26) ==== 74+0+18506 (secure 0 0 0) 0x7fe4bc0597c0 con 0x7fe4cc073590 2024-06-09T22:01:07.690 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-06-09T22:01:07.690 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":26,"fsid":"08d1d92e-3533-4051-bd41-97c73d8b6b19","created":"2024-06-09T22:00:27.411675+0000","modified":"2024-06-09T22:01:03.716360+0000","last_up_change":"2024-06-09T22:00:42.024015+0000","last_in_change":"2024-06-09T22:00:33.132294+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-09T22:00:41.893545+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-06-09T22:00:54.102789+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-06-09T22:00:55.454518+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-06-09T22:01:00.813566+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"4cc9d666-a633-4740-a293-be07ab208937","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6811","nonce":3275688150}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6813","nonce":3275688150}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6817","nonce":3275688150}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":3275688150},{"type":"v1","addr":"172.21.15.32:6815","nonce":3275688150}]},"public_addr":"172.21.15.32:6811/3275688150","cluster_addr":"172.21.15.32:6813/3275688150","heartbeat_back_addr":"172.21.15.32:6817/3275688150","heartbeat_front_addr":"172.21.15.32:6815/3275688150","state":["exists","up"]},{"osd":1,"uuid":"8db4805f-af3c-41a5-8eaf-1485c4ec8a5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6803","nonce":3057205177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6805","nonce":3057205177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6809","nonce":3057205177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":3057205177},{"type":"v1","addr":"172.21.15.32:6807","nonce":3057205177}]},"public_addr":"172.21.15.32:6803/3057205177","cluster_addr":"172.21.15.32:6805/3057205177","heartbeat_back_addr":"172.21.15.32:6809/3057205177","heartbeat_front_addr":"172.21.15.32:6807/3057205177","state":["exists","up"]},{"osd":2,"uuid":"172c9719-1b63-460d-ba3e-dba05f194990","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6819","nonce":3248807238}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6821","nonce":3248807238}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6825","nonce":3248807238}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":3248807238},{"type":"v1","addr":"172.21.15.32:6823","nonce":3248807238}]},"public_addr":"172.21.15.32:6819/3248807238","cluster_addr":"172.21.15.32:6821/3248807238","heartbeat_back_addr":"172.21.15.32:6825/3248807238","heartbeat_front_addr":"172.21.15.32:6823/3248807238","state":["exists","up"]},{"osd":3,"uuid":"611e5921-1524-4cc9-9f33-a51d431115f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6827","nonce":1835052153}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6829","nonce":1835052153}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6833","nonce":1835052153}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":1835052153},{"type":"v1","addr":"172.21.15.32:6831","nonce":1835052153}]},"public_addr":"172.21.15.32:6827/1835052153","cluster_addr":"172.21.15.32:6829/1835052153","heartbeat_back_addr":"172.21.15.32:6833/1835052153","heartbeat_front_addr":"172.21.15.32:6831/1835052153","state":["exists","up"]},{"osd":4,"uuid":"b85f7437-19ea-4f08-8a71-ef14330e0351","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6800","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6801","nonce":531504437}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6802","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6803","nonce":531504437}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6806","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6807","nonce":531504437}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6804","nonce":531504437},{"type":"v1","addr":"172.21.15.148:6805","nonce":531504437}]},"public_addr":"172.21.15.148:6801/531504437","cluster_addr":"172.21.15.148:6803/531504437","heartbeat_back_addr":"172.21.15.148:6807/531504437","heartbeat_front_addr":"172.21.15.148:6805/531504437","state":["exists","up"]},{"osd":5,"uuid":"ae110414-eb0e-4253-91f2-0b2e83dad115","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6808","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6810","nonce":2398614348}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6812","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6814","nonce":2398614348}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6820","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6822","nonce":2398614348}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6816","nonce":2398614348},{"type":"v1","addr":"172.21.15.148:6818","nonce":2398614348}]},"public_addr":"172.21.15.148:6810/2398614348","cluster_addr":"172.21.15.148:6814/2398614348","heartbeat_back_addr":"172.21.15.148:6822/2398614348","heartbeat_front_addr":"172.21.15.148:6818/2398614348","state":["exists","up"]},{"osd":6,"uuid":"0de55946-8987-4376-b9f9-f4957200232f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6809","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6811","nonce":975384648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6813","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6815","nonce":975384648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6821","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6823","nonce":975384648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6817","nonce":975384648},{"type":"v1","addr":"172.21.15.148:6819","nonce":975384648}]},"public_addr":"172.21.15.148:6811/975384648","cluster_addr":"172.21.15.148:6815/975384648","heartbeat_back_addr":"172.21.15.148:6823/975384648","heartbeat_front_addr":"172.21.15.148:6819/975384648","state":["exists","up"]},{"osd":7,"uuid":"0524e1f0-9c95-4d96-8d3f-e120b647bc83","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":24,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6824","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6825","nonce":2655085429}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6826","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6827","nonce":2655085429}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6830","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6831","nonce":2655085429}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.148:6828","nonce":2655085429},{"type":"v1","addr":"172.21.15.148:6829","nonce":2655085429}]},"public_addr":"172.21.15.148:6825/2655085429","cluster_addr":"172.21.15.148:6827/2655085429","heartbeat_back_addr":"172.21.15.148:6831/2655085429","heartbeat_front_addr":"172.21.15.148:6829/2655085429","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.620927+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.493643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.657004+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.616764+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.596137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.580349+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.602447+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-06-09T22:00:38.609775+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-09T22:01:07.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe4ac06be80 msgr2=0x7fe4ac06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:07.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe4ac06be80 0x7fe4ac06e340 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc1528b0 tx=0x7fe4b402e000 comp rx=0 tx=0).stop 2024-06-09T22:01:07.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc073590 msgr2=0x7fe4cc140360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-09T22:01:07.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc073590 0x7fe4cc140360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00e930 tx=0x7fe4bc00edf0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 shutdown_connections 2024-06-09T22:01:07.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.148:3300/0,v1:172.21.15.148:6789/0] conn(0x7fe4cc157690 0x7fe4cc0c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4cc150d70 0x7fe4cc0c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.32:6800/64084074,v1:172.21.15.32:6801/64084074] conn(0x7fe4ac06be80 0x7fe4ac06e340 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 --2- 172.21.15.32:0/2155546528 >> [v2:172.21.15.148:3301/0,v1:172.21.15.148:6790/0] conn(0x7fe4cc073590 0x7fe4cc140360 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-09T22:01:07.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 >> 172.21.15.32:0/2155546528 conn(0x7fe4cc0cbb70 msgr2=0x7fe4cc0cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-09T22:01:07.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 shutdown_connections 2024-06-09T22:01:07.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-06-09T22:01:07.708+0000 7fe4d2329640 1 -- 172.21.15.32:0/2155546528 wait complete. 2024-06-09T22:01:07.804 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 452, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 557, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 749, in create self.run_client_payload(f"setfattr -n ceph.dir.layout.pool -v {ec_data_pool_name} . && getfattr -n ceph.dir.layout .") File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 831, in run_client_payload self.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1045, in are_daemons_healthy mds.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c" 2024-06-09T22:01:07.806 INFO:teuthology.misc:Shutting down mds daemons... 2024-06-09T22:01:07.806 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-06-09T22:01:07.807 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:07.821 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-06-09T22:01:07.827+0000 b71a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i a (PID: 41596) UID: 0 2024-06-09T22:01:07.821 INFO:tasks.ceph.mds.a.smithi032.stderr:2024-06-09T22:01:07.828+0000 b71a640 -1 mds.a *** got signal Terminated *** 2024-06-09T22:01:11.826 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:01:11.829 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:01:13.907 INFO:tasks.ceph.mds.a:Stopped 2024-06-09T22:01:13.908 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-06-09T22:01:13.908 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-09T22:01:13.908 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 452, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 557, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 749, in create self.run_client_payload(f"setfattr -n ceph.dir.layout.pool -v {ec_data_pool_name} . && getfattr -n ceph.dir.layout .") File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 831, in run_client_payload self.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1045, in are_daemons_healthy mds.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c" 2024-06-09T22:01:13.908 INFO:tasks.ceph.mds.c:Stopped 2024-06-09T22:01:13.909 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-06-09T22:01:13.909 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:13.924 INFO:tasks.ceph.mds.e.smithi032.stderr:2024-06-09T22:01:13.928+0000 b71a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.e.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i e (PID: 41609) UID: 0 2024-06-09T22:01:13.924 INFO:tasks.ceph.mds.e.smithi032.stderr:2024-06-09T22:01:13.930+0000 b71a640 -1 mds.e *** got signal Terminated *** 2024-06-09T22:01:20.010 INFO:tasks.ceph.mds.e:Stopped 2024-06-09T22:01:20.011 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-06-09T22:01:20.011 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:20.024 INFO:tasks.ceph.mds.b.smithi148.stderr:2024-06-09T22:01:20.030+0000 b71a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i b (PID: 41550) UID: 0 2024-06-09T22:01:20.024 INFO:tasks.ceph.mds.b.smithi148.stderr:2024-06-09T22:01:20.031+0000 b71a640 -1 mds.b *** got signal Terminated *** 2024-06-09T22:01:26.112 INFO:tasks.ceph.mds.b:Stopped 2024-06-09T22:01:26.112 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-06-09T22:01:26.113 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:26.128 INFO:tasks.ceph.mds.d.smithi148.stderr:2024-06-09T22:01:26.133+0000 b71a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.d.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i d (PID: 41552) UID: 0 2024-06-09T22:01:26.128 INFO:tasks.ceph.mds.d.smithi148.stderr:2024-06-09T22:01:26.135+0000 b71a640 -1 mds.d *** got signal Terminated *** 2024-06-09T22:01:32.215 INFO:tasks.ceph.mds.d:Stopped 2024-06-09T22:01:32.215 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-06-09T22:01:32.215 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:32.229 INFO:tasks.ceph.mds.f.smithi148.stderr:2024-06-09T22:01:32.236+0000 b71a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.f.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i f (PID: 41555) UID: 0 2024-06-09T22:01:32.230 INFO:tasks.ceph.mds.f.smithi148.stderr:2024-06-09T22:01:32.237+0000 b71a640 -1 mds.f *** got signal Terminated *** 2024-06-09T22:01:38.317 INFO:tasks.ceph.mds.f:Stopped 2024-06-09T22:01:38.317 INFO:teuthology.misc:Shutting down osd daemons... 2024-06-09T22:01:38.318 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-06-09T22:01:38.318 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:38.319 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-06-09T22:01:38.335+0000 7f227cb95640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 0 (PID: 37913) UID: 0 2024-06-09T22:01:38.319 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-06-09T22:01:38.335+0000 7f227cb95640 -1 osd.0 33 *** Got signal Terminated *** 2024-06-09T22:01:41.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:01:41.888 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:01:44.420 INFO:tasks.ceph.osd.0:Stopped 2024-06-09T22:01:44.420 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-06-09T22:01:44.420 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:44.421 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-06-09T22:01:44.438+0000 7f8c60f00640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 1 (PID: 37917) UID: 0 2024-06-09T22:01:44.421 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-06-09T22:01:44.438+0000 7f8c60f00640 -1 osd.1 35 *** Got signal Terminated *** 2024-06-09T22:01:50.522 INFO:tasks.ceph.osd.1:Stopped 2024-06-09T22:01:50.523 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-06-09T22:01:50.523 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:50.524 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-06-09T22:01:50.540+0000 7fe0c7e5c640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 2 (PID: 37923) UID: 0 2024-06-09T22:01:50.524 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-06-09T22:01:50.540+0000 7fe0c7e5c640 -1 osd.2 37 *** Got signal Terminated *** 2024-06-09T22:01:56.625 INFO:tasks.ceph.osd.2:Stopped 2024-06-09T22:01:56.625 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-06-09T22:01:56.625 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:01:56.626 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-06-09T22:01:56.643+0000 7f4247965640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 3 (PID: 37926) UID: 0 2024-06-09T22:01:56.627 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-06-09T22:01:56.643+0000 7f4247965640 -1 osd.3 39 *** Got signal Terminated *** 2024-06-09T22:02:02.727 INFO:tasks.ceph.osd.3:Stopped 2024-06-09T22:02:02.728 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-06-09T22:02:02.728 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:02.729 INFO:tasks.ceph.osd.4.smithi148.stderr:2024-06-09T22:02:02.745+0000 7fd8cb587640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 4 (PID: 37949) UID: 0 2024-06-09T22:02:02.729 INFO:tasks.ceph.osd.4.smithi148.stderr:2024-06-09T22:02:02.745+0000 7fd8cb587640 -1 osd.4 41 *** Got signal Terminated *** 2024-06-09T22:02:08.830 INFO:tasks.ceph.osd.4:Stopped 2024-06-09T22:02:08.831 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-06-09T22:02:08.831 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:08.832 INFO:tasks.ceph.osd.5.smithi148.stderr:2024-06-09T22:02:08.848+0000 7f8f90941640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 5 (PID: 37951) UID: 0 2024-06-09T22:02:08.832 INFO:tasks.ceph.osd.5.smithi148.stderr:2024-06-09T22:02:08.848+0000 7f8f90941640 -1 osd.5 43 *** Got signal Terminated *** 2024-06-09T22:02:12.175 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:02:12.178 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:02:14.933 INFO:tasks.ceph.osd.5:Stopped 2024-06-09T22:02:14.934 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-06-09T22:02:14.934 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:14.935 INFO:tasks.ceph.osd.6.smithi148.stderr:2024-06-09T22:02:14.951+0000 7f9a17595640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 6 (PID: 37956) UID: 0 2024-06-09T22:02:14.935 INFO:tasks.ceph.osd.6.smithi148.stderr:2024-06-09T22:02:14.951+0000 7f9a17595640 -1 osd.6 45 *** Got signal Terminated *** 2024-06-09T22:02:21.036 INFO:tasks.ceph.osd.6:Stopped 2024-06-09T22:02:21.036 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-06-09T22:02:21.036 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:21.037 INFO:tasks.ceph.osd.7.smithi148.stderr:2024-06-09T22:02:21.053+0000 7fbc63afe640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-osd -f --cluster ceph -i 7 (PID: 37955) UID: 0 2024-06-09T22:02:21.037 INFO:tasks.ceph.osd.7.smithi148.stderr:2024-06-09T22:02:21.053+0000 7fbc63afe640 -1 osd.7 45 *** Got signal Terminated *** 2024-06-09T22:02:27.137 INFO:tasks.ceph.osd.7:Stopped 2024-06-09T22:02:27.138 INFO:teuthology.misc:Shutting down mgr daemons... 2024-06-09T22:02:27.138 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-06-09T22:02:27.138 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:27.139 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-06-09T22:02:27.155+0000 7fa703b60640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 37667) UID: 0 2024-06-09T22:02:27.167 INFO:tasks.ceph.mgr.y:Stopped 2024-06-09T22:02:27.167 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-06-09T22:02:27.168 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:27.195 INFO:tasks.ceph.mgr.x:Stopped 2024-06-09T22:02:27.196 INFO:teuthology.misc:Shutting down mon daemons... 2024-06-09T22:02:27.196 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-09T22:02:27.196 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:27.224 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-06-09T22:02:27.228+0000 a0c3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 37631) UID: 0 2024-06-09T22:02:27.224 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-06-09T22:02:27.230+0000 a0c3640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-09T22:02:27.934 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:12.447 37657== 2024-06-09T22:02:27.934 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:12.447 37657== Exit program on first error (--exit-on-first-error=yes) 2024-06-09T22:02:27.946 INFO:tasks.ceph.mon.a.smithi032.stderr:daemon-helper: command failed with exit status 42 2024-06-09T22:02:33.298 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-09T22:02:33.298 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 452, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 557, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 749, in create self.run_client_payload(f"setfattr -n ceph.dir.layout.pool -v {ec_data_pool_name} . && getfattr -n ceph.dir.layout .") File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 831, in run_client_payload self.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1045, in are_daemons_healthy mds.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-06-09T22:02:33.299 INFO:tasks.ceph.mon.a:Stopped 2024-06-09T22:02:33.299 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-09T22:02:33.300 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:33.315 INFO:tasks.ceph.mon.b.smithi148.stderr:2024-06-09T22:02:33.320+0000 a0c3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 37370) UID: 0 2024-06-09T22:02:33.317 INFO:tasks.ceph.mon.b.smithi148.stderr:2024-06-09T22:02:33.322+0000 a0c3640 -1 mon.b@1(peon) e1 *** Got Signal Terminated *** 2024-06-09T22:02:38.532 INFO:tasks.ceph.mon.b.smithi148.stderr:==00:00:02:22.994 37377== 2024-06-09T22:02:38.533 INFO:tasks.ceph.mon.b.smithi148.stderr:==00:00:02:22.994 37377== Exit program on first error (--exit-on-first-error=yes) 2024-06-09T22:02:38.543 INFO:tasks.ceph.mon.b.smithi148.stderr:daemon-helper: command failed with exit status 42 2024-06-09T22:02:39.400 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-09T22:02:39.401 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 452, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 557, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 749, in create self.run_client_payload(f"setfattr -n ceph.dir.layout.pool -v {ec_data_pool_name} . && getfattr -n ceph.dir.layout .") File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 831, in run_client_payload self.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1045, in are_daemons_healthy mds.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-06-09T22:02:39.401 INFO:tasks.ceph.mon.b:Stopped 2024-06-09T22:02:39.401 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-09T22:02:39.402 INFO:teuthology.orchestra.run:waiting for 300 2024-06-09T22:02:39.421 INFO:tasks.ceph.mon.c.smithi148.stderr:2024-06-09T22:02:39.422+0000 a0c3640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 37372) UID: 0 2024-06-09T22:02:39.421 INFO:tasks.ceph.mon.c.smithi148.stderr:2024-06-09T22:02:39.423+0000 a0c3640 -1 mon.c@2(probing) e1 *** Got Signal Terminated *** 2024-06-09T22:02:39.891 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:24.353 37378== 2024-06-09T22:02:39.891 INFO:tasks.ceph.mon.c.smithi148.stderr:==00:00:02:24.353 37378== Exit program on first error (--exit-on-first-error=yes) 2024-06-09T22:02:39.902 INFO:tasks.ceph.mon.c.smithi148.stderr:daemon-helper: command failed with exit status 42 2024-06-09T22:02:42.249 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:02:42.252 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:02:45.504 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-09T22:02:45.504 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 452, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 557, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 749, in create self.run_client_payload(f"setfattr -n ceph.dir.layout.pool -v {ec_data_pool_name} . && getfattr -n ceph.dir.layout .") File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 831, in run_client_payload self.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1045, in are_daemons_healthy mds.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi148 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-06-09T22:02:45.504 INFO:tasks.ceph.mon.c:Stopped 2024-06-09T22:02:45.505 INFO:tasks.ceph:Checking cluster log for badness... 2024-06-09T22:02:45.505 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v MON_DOWN | head -n 1 2024-06-09T22:02:45.547 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T22:02:45.547 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-06-09T22:02:45.663 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T22:02:45.664 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-06-09T22:02:45.759 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T22:02:45.760 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-06-09T22:02:45.837 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T22:02:45.837 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-06-09T22:02:45.911 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T22:02:45.911 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-06-09T22:02:46.095 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T22:02:46.095 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-06-09T22:02:46.217 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T22:02:46.218 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-06-09T22:02:46.267 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T22:02:46.267 DEBUG:teuthology.orchestra.run.smithi148:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-06-09T22:02:46.316 INFO:tasks.ceph:Archiving mon data... 2024-06-09T22:02:46.319 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/data/mon.a.tgz 2024-06-09T22:02:46.320 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-06-09T22:02:46.339 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.QqpJoRA0Ki 2024-06-09T22:02:46.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.QqpJoRA0Ki 2024-06-09T22:02:46.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.QqpJoRA0Ki 2024-06-09T22:02:46.573 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.QqpJoRA0Ki is 289KB 2024-06-09T22:02:46.654 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.QqpJoRA0Ki 2024-06-09T22:02:46.671 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/data/mon.b.tgz 2024-06-09T22:02:46.671 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-06-09T22:02:46.690 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.S4sp2nK7ZB 2024-06-09T22:02:46.690 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.S4sp2nK7ZB 2024-06-09T22:02:46.842 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.S4sp2nK7ZB 2024-06-09T22:02:46.928 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.S4sp2nK7ZB is 388KB 2024-06-09T22:02:47.024 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.S4sp2nK7ZB 2024-06-09T22:02:47.040 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/data/mon.c.tgz 2024-06-09T22:02:47.040 DEBUG:teuthology.orchestra.run.smithi148:> mktemp 2024-06-09T22:02:47.095 INFO:teuthology.orchestra.run.smithi148.stdout:/tmp/tmp.bEgZiKHB07 2024-06-09T22:02:47.095 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.bEgZiKHB07 2024-06-09T22:02:47.231 DEBUG:teuthology.orchestra.run.smithi148:> sudo chmod 0666 /tmp/tmp.bEgZiKHB07 2024-06-09T22:02:47.316 DEBUG:teuthology.orchestra.remote:smithi148:/tmp/tmp.bEgZiKHB07 is 388KB 2024-06-09T22:02:47.413 DEBUG:teuthology.orchestra.run.smithi148:> rm -fr /tmp/tmp.bEgZiKHB07 2024-06-09T22:02:47.429 INFO:tasks.ceph:Cleaning ceph cluster... 2024-06-09T22:02:47.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-09T22:02:47.431 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-09T22:02:47.703 DEBUG:tasks.ceph:valgrind exception message: valgrind error: UninitCondition UnknownInlinedFun UnknownInlinedFun QuiesceDbManager::db_thread_has_work() const 2024-06-09T22:02:47.942 INFO:tasks.ceph:Archiving crash dumps... 2024-06-09T22:02:47.944 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/remote/smithi032/crash 2024-06-09T22:02:47.945 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-09T22:02:47.975 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/remote/smithi148/crash 2024-06-09T22:02:47.976 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-09T22:02:48.001 INFO:tasks.ceph:Shutting down logrotate 2024-06-09T22:02:48.001 DEBUG:teuthology.orchestra.run.smithi032:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:02:48.020 DEBUG:teuthology.orchestra.run.smithi148:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-06-09T22:02:48.088 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-06-09T22:02:48.117 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-06-09T22:02:48.141 INFO:tasks.ceph:Compressing logs... 2024-06-09T22:02:48.141 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-09T22:02:48.160 DEBUG:teuthology.orchestra.run.smithi148:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-06-09T22:02:48.186 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37670.log 2024-06-09T22:02:48.192 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31630.log 2024-06-09T22:02:48.192 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37670.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42422.log 2024-06-09T22:02:48.192 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.tmp-client.admin.31630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42061.log 2024-06-09T22:02:48.192 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31630.log.gz 2024-06-09T22:02:48.192 INFO:teuthology.orchestra.run.smithi032.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.37670.log.gz 2024-06-09T22:02:48.192 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42422.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-09T22:02:48.192 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42061.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42422.log.gz 2024-06-09T22:02:48.192 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41973.log 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42061.log.gz 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41772.log 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41889.log 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr: 84.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.41973.log.gz 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41772.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41772.log.gz 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38123.log 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.2.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41889.log.gz 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41614.log 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38123.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-09T22:02:48.193 INFO:teuthology.orchestra.run.smithi032.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.38123.log.gz 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.41614.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42295.log 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41614.log.gz 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37938.log 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42295.log.gz 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr: 89.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41499.log 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37938.log.gz 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42169.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42211.log 2024-06-09T22:02:48.194 INFO:teuthology.orchestra.run.smithi032.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41499.log.gz 2024-06-09T22:02:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-06-09T22:02:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.42211.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-06-09T22:02:48.206 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42211.log.gz 2024-06-09T22:02:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41702.log 2024-06-09T22:02:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-06-09T22:02:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mgr.y.log: /var/log/ceph/ceph-client.admin.41702.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42337.log 2024-06-09T22:02:48.206 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41702.log.gz 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.e.log 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42337.log: gzip -5 --verbose -- 89.1% /var/log/ceph/valgrind/mon.a.log 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.42337.log.gz 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mds.e.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.c.log 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr: 57.5% -- replaced with /var/log/ceph/valgrind/mds.e.log.gz 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.a.log 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr: 69.7%/var/log/ceph/valgrind/mds.c.log: -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41931.log 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr: 66.7% -- replaced with /var/log/ceph/valgrind/mds.c.log.gz 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mds.a.log: gzip/var/log/ceph/ceph-client.admin.41931.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.42108.log 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mon.a.log: 57.3% 88.2% -- replaced with /var/log/ceph/valgrind/mds.a.log.gz -- replaced with /var/log/ceph/ceph-client.admin.41931.log.gz 2024-06-09T22:02:48.207 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41847.log 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42108.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42015.log 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42108.log.gz 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41847.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42015.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41847.log.gz 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42015.log.gz 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42379.log 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42253.log 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr: 85.4% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42379.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42253.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.42379.log.gz 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.42253.log.gz 2024-06-09T22:02:48.208 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mds.a.log: 84.7% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-06-09T22:02:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37660.log 2024-06-09T22:02:48.218 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:gzip/var/log/ceph/ceph-client.admin.37660.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.37780.log 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37620.log 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.37660.log.gz 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37780.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.37780.log.gz 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37620.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr: 91.5% -- replaced with /var/log/ceph/ceph.log.gz 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.37620.log.gz 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37571.log 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr: 87.2% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37571.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37571.log.gz 2024-06-09T22:02:48.219 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37820.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mds.b.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37820.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr: 86.2% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37820.log.gz 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37700.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37700.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37700.log.gz 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.d.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mds.f.log 2024-06-09T22:02:48.220 INFO:teuthology.orchestra.run.smithi148.stderr: 69.7%/var/log/ceph/valgrind/mds.d.log: gzip -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr: 57.5% -- replaced with /var/log/ceph/valgrind/mds.d.log.gz 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:gzip/var/log/ceph/valgrind/mds.f.log: -5 --verbose -- /var/log/ceph/valgrind/mds.b.log 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr: 57.5% -- replaced with /var/log/ceph/valgrind/mds.f.log.gz 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 69.7% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/valgrind/mds.b.log: gzip 57.5% -- replaced with /var/log/ceph/valgrind/mds.b.log.gz 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.37860.log 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37740.log 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37860.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.37860.log.gz 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-client.admin.37740.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.37740.log.gz 2024-06-09T22:02:48.221 INFO:teuthology.orchestra.run.smithi148.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-06-09T22:02:48.222 INFO:teuthology.orchestra.run.smithi148.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: 86.2% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-06-09T22:02:48.225 INFO:teuthology.orchestra.run.smithi032.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-06-09T22:02:48.275 INFO:teuthology.orchestra.run.smithi148.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-06-09T22:02:48.290 INFO:teuthology.orchestra.run.smithi148.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-06-09T22:02:48.343 INFO:teuthology.orchestra.run.smithi032.stderr: 90.0% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-06-09T22:02:48.737 INFO:teuthology.orchestra.run.smithi032.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-06-09T22:02:48.745 INFO:teuthology.orchestra.run.smithi148.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-06-09T22:02:48.751 INFO:teuthology.orchestra.run.smithi148.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-06-09T22:02:48.763 INFO:teuthology.orchestra.run.smithi032.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-06-09T22:02:48.772 INFO:teuthology.orchestra.run.smithi032.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-06-09T22:02:48.791 INFO:teuthology.orchestra.run.smithi148.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-06-09T22:02:48.800 INFO:teuthology.orchestra.run.smithi148.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-06-09T22:02:48.802 INFO:teuthology.orchestra.run.smithi148.stderr: 2024-06-09T22:02:48.802 INFO:teuthology.orchestra.run.smithi148.stderr:real 0m0.606s 2024-06-09T22:02:48.803 INFO:teuthology.orchestra.run.smithi148.stderr:user 0m2.314s 2024-06-09T22:02:48.803 INFO:teuthology.orchestra.run.smithi148.stderr:sys 0m0.120s 2024-06-09T22:02:48.803 INFO:teuthology.orchestra.run.smithi032.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-06-09T22:02:48.805 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-06-09T22:02:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m0.631s 2024-06-09T22:02:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m2.429s 2024-06-09T22:02:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m0.118s 2024-06-09T22:02:48.806 INFO:tasks.ceph:Archiving logs... 2024-06-09T22:02:48.806 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/remote/smithi032/log 2024-06-09T22:02:48.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-09T22:02:49.272 DEBUG:teuthology.misc:Transferring archived files from smithi148:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/remote/smithi148/log 2024-06-09T22:02:49.273 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-09T22:02:49.697 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 452, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 557, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 749, in create self.run_client_payload(f"setfattr -n ceph.dir.layout.pool -v {ec_data_pool_name} . && getfattr -n ceph.dir.layout .") File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 831, in run_client_payload self.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1045, in are_daemons_healthy mds.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: UninitCondition UnknownInlinedFun UnknownInlinedFun QuiesceDbManager::db_thread_has_work() const 2024-06-09T22:02:49.885 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7cf0803bb6294e69a096cd879aeea8bf Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 452, in cephfs_setup fs = Filesystem(ctx, fs_config=temp, name=name, create=True) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 557, in __init__ self.create() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 749, in create self.run_client_payload(f"setfattr -n ceph.dir.layout.pool -v {ec_data_pool_name} . && getfattr -n ceph.dir.layout .") File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 831, in run_client_payload self.wait_for_daemons() File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1240, in wait_for_daemons if self.are_daemons_healthy(status=status, skip_max_mds_check=skip_max_mds_check): File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/cephfs/filesystem.py", line 1045, in are_daemons_healthy mds.check_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 43, in check_status return self.proc.poll() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 211, in poll self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mds.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-mds -f --cluster ceph -i c" During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_7b7a5d375cf4e4afab96b9d4d229b699ae8304bb/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: UninitCondition UnknownInlinedFun UnknownInlinedFun QuiesceDbManager::db_thread_has_work() const 2024-06-09T22:02:49.891 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-06-09T22:02:49.902 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-09T22:02:49.927 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-09T22:02:49.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-09T22:02:49.941 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-09T22:02:50.009 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-09T22:02:50.020 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-09T22:02:50.021 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T22:02:50.024 DEBUG:teuthology.orchestra.run.smithi148:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-09T22:02:50.039 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-06-09T22:02:50.041 INFO:teuthology.orchestra.run.smithi148.stderr:bash: line 1: ntpq: command not found 2024-06-09T22:02:50.045 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T22:02:50.045 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-06-09T22:02:50.045 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv01.front.sepia.ceph.com 3 6 377 40 +808us[ +806us] +/- 54ms 2024-06-09T22:02:50.045 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 38 -730us[ -730us] +/- 67ms 2024-06-09T22:02:50.045 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -368us[ -368us] +/- 76ms 2024-06-09T22:02:50.045 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:02:50.047 INFO:teuthology.orchestra.run.smithi148.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-09T22:02:50.047 INFO:teuthology.orchestra.run.smithi148.stdout:=============================================================================== 2024-06-09T22:02:50.048 INFO:teuthology.orchestra.run.smithi148.stdout:^* hv01.front.sepia.ceph.com 3 6 377 37 +802us[ +797us] +/- 54ms 2024-06-09T22:02:50.048 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 36 -746us[ -746us] +/- 67ms 2024-06-09T22:02:50.048 INFO:teuthology.orchestra.run.smithi148.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 -389us[ -389us] +/- 76ms 2024-06-09T22:02:50.048 INFO:teuthology.orchestra.run.smithi148.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-09T22:02:50.048 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-09T22:02:50.060 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-09T22:02:50.060 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-09T22:02:50.073 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-09T22:02:50.157 DEBUG:teuthology.orchestra.run.smithi148:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-09T22:02:50.225 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T22:02:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:02:50.256 DEBUG:teuthology.orchestra.run.smithi148:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-06-09T22:02:50.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:02:50.286 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-09T22:02:50.296 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-09T22:02:50.307 INFO:teuthology.task.internal:Duration was 960.490697 seconds 2024-06-09T22:02:50.307 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-09T22:02:50.320 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-09T22:02:50.320 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-09T22:02:50.323 DEBUG:teuthology.orchestra.run.smithi148:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-09T22:02:50.363 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T22:02:50.367 INFO:teuthology.orchestra.run.smithi148.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-09T22:02:50.801 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-09T22:02:50.801 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-06-09T22:02:50.801 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-09T22:02:50.823 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi148.front.sepia.ceph.com 2024-06-09T22:02:50.824 DEBUG:teuthology.orchestra.run.smithi148:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-09T22:02:50.867 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-09T22:02:50.868 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-09T22:02:50.870 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-09T22:02:50.936 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-09T22:02:50.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-09T22:02:50.941 DEBUG:teuthology.orchestra.run.smithi148:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-09T22:02:51.074 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-09T22:02:51.085 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-09T22:02:51.085 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-09T22:02:51.123 DEBUG:teuthology.orchestra.run.smithi148:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-09T22:02:51.149 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-09T22:02:51.161 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:02:51.166 DEBUG:teuthology.orchestra.run.smithi148:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:02:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-06-09T22:02:51.212 INFO:teuthology.orchestra.run.smithi148.stdout:kernel.core_pattern = core 2024-06-09T22:02:51.327 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:02:51.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:02:51.383 DEBUG:teuthology.orchestra.run.smithi148:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-09T22:02:51.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-09T22:02:51.398 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-09T22:02:51.408 INFO:teuthology.task.internal:Transferring archived files... 2024-06-09T22:02:51.409 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/remote/smithi032 2024-06-09T22:02:51.409 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-09T22:02:51.459 DEBUG:teuthology.misc:Transferring archived files from smithi148:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-09_21:24:02-fs-squid-distro-default-smithi/7747204/remote/smithi148 2024-06-09T22:02:51.460 DEBUG:teuthology.orchestra.run.smithi148:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-09T22:02:51.493 INFO:teuthology.task.internal:Removing archive directory... 2024-06-09T22:02:51.493 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-09T22:02:51.496 DEBUG:teuthology.orchestra.run.smithi148:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-09T22:02:51.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-09T22:02:51.556 INFO:teuthology.task.internal:Not uploading archives. 2024-06-09T22:02:51.556 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-09T22:02:51.572 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-09T22:02:51.572 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-09T22:02:51.575 DEBUG:teuthology.orchestra.run.smithi148:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-09T22:02:51.588 INFO:teuthology.orchestra.run.smithi032.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 9 22:02 /home/ubuntu/cephtest 2024-06-09T22:02:51.600 INFO:teuthology.orchestra.run.smithi148.stdout: 265316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 9 22:02 /home/ubuntu/cephtest 2024-06-09T22:02:51.601 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-09T22:02:51.613 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-09T22:02:51.639 INFO:teuthology.run:Summary data: description: fs/verify/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore-ec/bluestore-comp-ec-root overrides/{ignorelist_health ignorelist_wrongly_marked_down mon-debug pg_health session_timeout} ranks/1 tasks/dbench validater/valgrind} duration: 960.4906966686249 failure_reason: 'valgrind error: UninitCondition UnknownInlinedFun UnknownInlinedFun QuiesceDbManager::db_thread_has_work() const' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7cf0803bb6294e69a096cd879aeea8bf status: fail success: false 2024-06-09T22:02:51.639 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-09T22:02:51.742 INFO:teuthology.run:FAIL